DroPE: Extending the Context Window of LLMs

Published on

An interesting paper came out from Sakana AI (the Japanese AI company that just signed a partnership with Google), proposing an LLM training technique called DroPE.

The idea goes after one of the biggest problems with LLMs: the context window. Everybody wants the model to summarize texts or understand giant codebases, but running training/fine-tuning with long sequences is absurdly expensive and slow (attention is quadratic, so the cost explodes).

Usually, a technique called RoPE (Rotary Positional Embeddings) is used to tell the model the order of the words. It works by applying rotations to the embedding vectors used in the attention mechanism, and during training, RoPE is fantastic. The model kind of learns to recognize those rotations and understands that if one word rotated “X” degrees and another “Y”, they’re a certain distance apart. That creates an ordering and helps the model learn faster.

The problem comes later, when you want to go beyond the context the model was trained on. Say it was trained on texts of up to 4k tokens, and you feed it one with 10k. At the new positions (4001 onward), RoPE will generate rotations the model has never seen in its life, and the model won’t really know how to interpret them. Since the new rotations look weird, the math doesn’t add up. The model produces bizarre attention scores and that messes up the quality of its attention.

There are techniques to try to scale RoPE (like YaRN, NTK scaling, etc.), which try to “adjust” those rotations to fit larger windows. But in practice, they still end up needing fine-tuning and can change the way the model distributes its attention (for example, getting more “hooked” on recent tokens instead of looking for information at the beginning or middle of the text).

The trick behind DroPE is to treat those RoPE positional embeddings as just the scaffolding of a construction. They’re essential for “building” (training) the model, but once the building is done, the scaffolding only blocks the view. Sticking with the analogy, it’s something like saying: “Now that you know how to read and write, focus only on the content of the words and ignore those rotations that are confusing you”.

So here’s what they proposed:

1 - Train the model normally with RoPE 2 - Completely remove the positional embeddings (the scaffolding) 3 - Do a quick, cheap recalibration

DroPE

And in the results, they saw that the models work much better on long contexts, beating techniques like YaRN. In the “Needle In A Haystack” tests, DroPE managed to stay focused on the right information, while the other methods got lost.

They tested it on models from the SmolLM family and even on the 7B Llama-2, and the performance gain was great, spending a tiny fraction of the original training budget on the adaptation (on SmolLM, they got close to the original performance spending less than ~1% of the token budget).

Here’s the link to the paper.