Byte language models: long sequences as a resource, not a burden

A tokenizer has long seemed an obligatory part of a language model: it splits text into meaningful chunks (subwords), shortening the sequence. Working on raw bytes looked wasteful — one subword spans roughly four bytes on average, so a byte model must process several times more positions.

A new paper by Jie Wang, Shiwei Luo (East China Normal University) and colleagues argues the opposite. The authors take an ordinary "flat" Transformer — with none of the hierarchical machinery of architectures like BLT or H-Net — and train it directly on bytes. Their claim: the extra sequence length is not a defect but additional computation available under the same parameter budget.

What the experiments show

With two ingredients — token-superposition training (averaging embeddings of four-token bags) and hash embeddings that fold in information about nearby n-grams — byte models consistently reach lower loss than subword models as parameters scale. The picture is not one-sided, though: at a matched compute (FLOPs) budget the subword model still wins. But the byte optima "prefer" 1.5–3 times more data per parameter, promising gains if the training corpus keeps growing.

On tasks that demand fine-grained perception of text the advantage is clear: about 40% relative improvement on CUTE (letter- and word-level manipulation) and some 20% on OCRBench.

Abstractions without a tokenizer

The most intriguing finding is that the model invents for itself what a tokenizer normally supplies. On a string-copying task, attention in the intermediate layers concentrates not uniformly but on a small set of boundary positions that gather local context — and this happens even with no whitespace in the input. To test how robust these internal structures are, the authors froze up to a quarter of the intermediate layers, forcing them to operate only on those aggregated representations; downstream performance held up, with no retraining. The same intervention wrecks a subword model.

Finally, prediction difficulty in the byte model is strikingly non-uniform: most bytes are predicted almost effortlessly, while uncertainty piles up near the boundaries of the learned structures. That suits speculative decoding — a small draft model here gets 3.4 times more tokens accepted than in the subword case.

The authors' conclusion is simple: sequence length, learned abstraction, and the allocation of computation should be treated as coupled design axes for future models.

Source: Lobsters