Polish 1 X 0 English: the most efficient language for LLMs

Published on

It sounds fake, but it’s not: English is NOT the best language for LLMs!

A really interesting paper came out presenting “ONERULER”, a new benchmark to evaluate LLMs on long contexts (up to 128k tokens) in 26 languages. And the results were, to say the least, counterintuitive.

We always assume that, since the overwhelming majority of these LLMs’ pre-training data is in English, it would be by far the best language on any task. But in the ONERULER ranking, English came in only 6th place.

And the gold medal went to my beloved Polish! (funnily enough, my 2nd favorite language among the ones I speak, right behind Italian)

But why on earth Polish?

According to the researchers, it was probably a matter of tokenization and “information density”. It seems the Slavic languages (Russian and Ukrainian also did super well) are in a “sweet spot” where the way the language gets compressed into tokens helps the model retain context more efficiently.

For those who don’t know, Slavic languages, in general, don’t conjugate only verbs, but also nouns and adjectives. This is called “declension”. Because of that, a word’s suffix changes according to its syntactic function in the sentence.

Example: the word “cat” in the Polish dictionary is “kot”, and the word “rat” is “szczur” (pronounced “sh-tchur”).

If a word is the subject of the sentence (doing an action), nothing changes; but if it’s the direct object (receiving an action), a suffix is added:

  • Kot zjadł szczura. (The cat ate the rat.)

Notice the “-a” suffix? In other words, the word carries within itself the information that it’s part of the direct object of the sentence. And because of that, word order doesn’t matter much. In English, swapping the order to “The rat ate the cat” changes the meaning of the sentence. In Polish, as long as you keep the suffix on whoever receives the action, saying “Kot zjadł szczura” or “Szczura zjadł kot” means the same thing.

If you wanted to say the rat ate the cat, it would be: “Szczur zjadł kota”, that is, the rat goes back to the dictionary form and the “-a” suffix moves to the end of “kot”.

Here, I only gave the example of the direct object (“accusative” case), but there are many other cases:

  • if I give a toy to the cat (indirect object, “dative” case), it changes to “kotu”
  • if I do something with the cat (“instrumental” case), it changes to “kotem”
  • if I talk about the cat (“locative” case), it changes to “kocie”

Declension table for the Polish word 'kot'

Etc. That’s what they meant by “information density” in tokenization. You can extract A LOT of information about the syntax and meaning of the sentence with fewer tokens. Besides, Polish, unlike Russian or Ukrainian, uses the Latin alphabet, which is used by English and other high-resource languages.

This also became evident when Chinese, a language with lots of resources on the internet, ended up among the 4 worst. Its writing system is logographic, and the tokenization is quite different.

Well, sorry for the little Polish lesson! 😅

Here’s the link to the paper.