Byte Latent Transformer (BLT): Patches are more scalable than tokens

Published on

Recently, the FAIR team at Meta released a really interesting paper, the Byte Latent Transformer (BLT), which I explained on the “IA Sob Controle” podcast, but I figured it was worth a post too, for those who prefer to take in this content as text. This new architecture aims to break away from the limitations of traditional tokenization and increase the efficiency, scalability and robustness of language models. So let’s get into how it works!


What is BLT?

Imagine you’re learning a new language. You might start by memorizing whole words, like a tourist with a phrasebook. That’s (simplifying a lot) the traditional approach of Language Models (LLMs), which use tokenization to split text into chunks, like words or subwords (tokens), and then learn to work with them. It works, but what if the sentence is written with mistakes, like “wher is the bathroom” or “wheres my umbrela?”? A traditional model could fail on some of those cases.

Now, imagine that instead of memorizing whole words, you learn the bytes that make up the alphabet, the sounds of each letter and how they combine to form syllables and, finally, words. That’s basically the idea behind the Byte Latent Transformer (BLT), a new architecture that abandons tokenization and learns from the text’s raw bytes.


Why Bytes and Not Words?

In computers, everything, from text in different languages to images and code, gets translated into sequences of bytes. By working directly with those fundamental building blocks, BLT is able to:

  1. Understand Deeply: Like a student who knows the alphabet, BLT can understand variations and mistakes. “wher” is still understandable, because it knows how that snippet and the ones around it are represented in bytes. It isn’t stuck with a fixed vocabulary of tokens.
  2. Be More Efficient: Imagine you need to assemble a piece of furniture. With the traditional approach (tokens), you’d have big, prefabricated pieces (words). That’s fast at first, but if a piece is missing or different, you can run into trouble. With BLT, you have a set of screws, nuts and small boards (bytes). Assembling might take a little longer, but you have way more flexibility and can build anything, adapting to different situations.
  3. Learn Subtle Relationships: BLT can pick up nuances that token-based models miss. Think about the difference between “present” (a gift) and “present” (being in a place). A token-based model might mix the two up, but BLT, by looking at the bytes, can understand the subtle differences in context.

How Does BLT Do This? The Magic of Dynamic Patches

The secret of BLT is in how it organizes the bytes. Instead of fixed tokens, it uses dynamic patches. Imagine you’re reading a book and, instead of reading sentences at fixed intervals, you read them according to how complex the text is. A simple, straightforward part? You read several sentences at once. A dense passage full of new information? You read the sentences more slowly, paying more attention.

That’s what BLT does with patches. It analyzes the text’s entropy, a measure of uncertainty or surprise. If the byte sequence is predictable (low entropy), BLT creates a long patch, processing more bytes at once. If the sequence is unpredictable (high entropy), signaling something new or complex, BLT creates a short patch, dedicating more computing power to understand that stretch. Something like:

  • Simple Text (low entropy): “The cat sleeps on the couch with its paw on its mouth.” - Long Patch (reads the whole sentence at once).
  • Complex Text (high entropy): “The irreducible complexity of quantum theory may present…” - Short Patch (reads the sentence slowly, splitting the words up more).

BLT Architecture

BLT works like an orchestra, with three main components working in harmony:

  1. Local Encoder: Like the musicians reading the sheet music, this component converts the raw bytes into initial representations, the patches. It’s lightweight and fast, focused on processing the individual bytes.
  2. Latent Global Transformer: The conductor of the orchestra. It takes the patch representations from the local encoder and, with its “global view”, understands the relationships between the different parts of the text, capturing the context and the deeper meaning.
  3. Local Decoder: The musicians who turn the notes on the sheet into music. This component takes the Global Transformer’s final representation and translates it into the byte sequence that forms the output text.

Results

The tests with BLT show it’s not just an interesting idea, but a powerful architecture. On several tasks, it matches or even beats traditional models like Llama 3, with important advantages:

  • Efficiency: In some cases, BLT uses 50% fewer FLOPS (a measure of computing power) than Llama 3 to do the same task. That means it’s faster and cheaper.
  • Robustness: BLT handles spelling mistakes, slang and language variations better, showing a deeper understanding of language.
  • Scalability: Increasing the model size and the patch size at the same time reduces the computational cost, while keeping or improving performance.
  • Character Understanding: It understands the nuances of each character, which is crucial for tasks like translating between languages with different alphabets.

Maybe the most exciting thing is BLT’s potential to redefine how we think about AI. It’s not just an incremental evolution, it aims to be a new paradigm, one that challenges the traditional tokenization model and brings a more adaptive and efficient approach. Ever imagined how this could impact applications like automatic translators or chatbots?

For those who like to explore the technical details or even experiment, the code is already available on GitHub. I recommend taking a look!

Now, a question for you all: what do you think about this transition from fixed tokenization to dynamic bytes? Do you see applications for this technology? Leave your ideas in the comments. It’s always a pleasure to trade ideas about these innovations!