Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.<br><a href=\"https://cdn-uploads.huggingface.co/production/uploads/626d1e1e72169e781945bf44/7Rn3dW-d0MqR2hvRdUOfI.gif\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/626d1e1e72169e781945bf44/7Rn3dW-d0MqR2hvRdUOfI.gif\" alt=\"fig1_architecture\"></a></p>\n","updatedAt":"2026-08-19T01:50:39.293Z","author":{"_id":"626d1e1e72169e781945bf44","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/626d1e1e72169e781945bf44/VKbdVYvXe9_MdvwTBw8SC.jpeg","fullname":"Abraham Owodunni","name":"Owos","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8697629570960999},"editors":["Owos"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/626d1e1e72169e781945bf44/VKbdVYvXe9_MdvwTBw8SC.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.15454","authors":[{"_id":"6a850a3c536bdd3bdd48f6f4","name":"Abraham Toluwase Owodunni","hidden":false},{"_id":"6a850a3c536bdd3bdd48f6f5","name":"Chibuzor Okocha","hidden":false},{"_id":"6a850a3c536bdd3bdd48f6f6","name":"Christan Grant","hidden":false},{"_id":"6a850a3c536bdd3bdd48f6f7","name":"Tomasz Limisiewicz","hidden":false},{"_id":"6a850a3c536bdd3bdd48f6f8","name":"Sachin Kumar","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/626d1e1e72169e781945bf44/vm3vU-iNLGF5UyNGLntet.gif"],"publishedAt":"2026-08-16T00:00:00.000Z","submittedOnDailyAt":"2026-08-19T00:00:00.000Z","title":"Dynamic Multi-Byte Prediction With Hierarchical Language Models","submittedOnDailyBy":{"_id":"626d1e1e72169e781945bf44","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/626d1e1e72169e781945bf44/VKbdVYvXe9_MdvwTBw8SC.jpeg","isPro":false,"fullname":"Abraham Owodunni","user":"Owos","type":"user","name":"Owos"},"summary":"Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.","upvotes":2,"discussionId":"6a850a3d536bdd3bdd48f6f9","githubRepo":"https://github.com/skai-research/lca-multibyte","githubRepoAddedBy":"user","ai_summary":"Multi-byte prediction accelerates byte-level hierarchical language models by generating parallel bytes via variable-length windows and causal attention masking, improving inference speed with minimal quality loss.","ai_keywords":["byte-level hierarchical language models","multi-byte prediction","multi-token prediction","variable-length prediction window","attention-masking","causal attention","inference throughput"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":0,"organization":{"_id":"64f00e771205f43449300336","name":"ohiostate","fullname":"The Ohio State University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/64f00dab77777df14e39d0b4/suMjJQTVx2mnCW9u3vahz.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"626d1e1e72169e781945bf44","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/626d1e1e72169e781945bf44/VKbdVYvXe9_MdvwTBw8SC.jpeg","isPro":false,"fullname":"Abraham Owodunni","user":"Owos","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"64f00e771205f43449300336","name":"ohiostate","fullname":"The Ohio State University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/64f00dab77777df14e39d0b4/suMjJQTVx2mnCW9u3vahz.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.15454.md","query":{}}">
Dynamic Multi-Byte Prediction With Hierarchical Language Models
Abstract
Multi-byte prediction accelerates byte-level hierarchical language models by generating parallel bytes via variable-length windows and causal attention masking, improving inference speed with minimal quality loss.
Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.
Community
Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.15454 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.15454 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.15454 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.