The History and Evolution of NLP (Natural Language Processing): a Narrative Timeline

Published on

When you do a Google search, use an online translator or chat with a virtual assistant, you rarely stop to think about what’s going on behind the scenes. But the truth is that behind these seemingly simple interactions lies one of the most fascinating areas of artificial intelligence: Natural Language Processing (NLP). It’s the discipline that tries to teach machines to understand, generate and manipulate human language, a challenge that has been tackled for decades, always bumping into the quirks and complexities of how we speak and write. After all, language isn’t just a set of cold rules: it’s loaded with context, ambiguity, irony and even mistakes that humans interpret effortlessly, but that machines need real mathematical juggling to process.

In this timeline, we’re going to travel from the very beginnings of NLP, back in the 1950s, when computers could barely line up two words, through the era of handwritten rules, the rise of statistical models, the deep learning revolution and, finally, the era of the Transformers: the giant language models that today impress (and sometimes scare) us with their ability to generate almost human text. Each advance was a step toward a bold goal: making machines truly master language. But how did we get here? And what challenges shaped the way? The journey starts here.

The 1950s and 1960s - The First Steps

Picture the early days of computing, back in the 1950s. Computers filled entire rooms and were slow compared to a simple cell phone today. Even so, some pioneers were already dreaming of making these machines understand human language. In 1950, the mathematician Alan Turing asked a provocative question: “Can machines think?” and proposed the famous Turing Test, in which a machine would try to hold a conversation as if it were human. That test became almost a legend in the field, a distant goal to be reached. In the following years, scientists started racking their brains: how do you teach natural human language to a machine that only knows 0s and 1s?

Back then, the dominant approach was teaching through explicit rules, almost like school grammar. Remember those syntax classes, with subject, verb and predicate? Well, computers of that era learned the same way: through hand-written linguistic rules. If the sentence had a certain structure, you applied a certain rule; if a certain word showed up, you looked it up in internal dictionaries. One of the first ambitious projects was machine translation. In 1954 there was a famous experiment: researchers in the US programmed an IBM 701 to translate simple sentences from Russian into English. Using a set of rules and a limited vocabulary (just a few hundred words), the machine managed to translate about 60 sentences. It seemed like magic (imagine, a machine translating languages!). The success was so hyped that a lot of people figured we’d have perfect automatic translation within a few years. Little did they know that human language is a much trickier beast…

Over time, it became clear that stuffing the machine with grammar rules wasn’t exactly the dream path. First, because every language has exceptions and nuances that don’t fit in simple tables. Second, because as soon as a sentence got a little more complex or different from what was expected, the program got tangled up. A famous 1966 report (the ALPAC report) cooled everyone’s enthusiasm, pointing out that machine translation was still far from being truly useful, despite all the effort. Funding for translation research shrank, and many realized the strategy needed a rethink.

Still, the 60s weren’t all failures. Some interesting systems appeared that imitated conversation, even if in a very superficial way. In 1966, a program called ELIZA was created at MIT by Joseph Weizenbaum. ELIZA simulated a Rogerian psychotherapist (the kind that bounces your question right back at the patient) using simple word-substitution tricks. If you said “I am sad”, the program followed a little rule and asked: “Why are you sad?”. There was no real understanding, it was all just a shell: ELIZA looked for keywords and put together canned answers. Even so, many people talking to it for the first time found the conversation “strangely human” and opened up to the program, believing it really understood their problems. It was a landmark moment, and kind of funny too: a machine so dumb in terms of comprehension, yet able to fool humans for a few minutes. These first steps, like translation, grammar rules and primitive chatbots, paved the way, showing both the potential and the immense challenges of the field of Natural Language Processing (NLP).

The 1970s and 1980s - The First Statistical Models

Entering the 1970s and 80s, the AI crowd started to realize that maybe it was time to try a different approach to NLP. Hand-written rules were hitting their limits, because natural languages have an endless supply of exceptions and contexts. It’s that old story: you can teach all the rules, but sooner or later a sentence shows up that triggers some exceptional case (“The bandage was wound around the wound”, with the same word used twice and meaning two different things) and the model gets completely lost. That’s when some researchers had a lightbulb moment: what if, instead of dictating all the rules, we made the machine learn on its own from data? In other words, give the computer lots of example sentences and texts and let it work out the “rules” or patterns itself, in a probabilistic way. That was the seed of the first statistical models for natural language.

One of the pioneering areas in this movement was speech recognition and part-of-speech tagging (automatically identifying nouns, verbs, etc., in a text). Researchers began applying Markov Chains and Hidden Markov Models (HMMs) to these tasks. Instead of coding every linguistic rule, these models calculated the probabilities of a sequence of words or sounds. For example, without anyone explicitly programming it, the model could “learn” that after the words “I want” the probability of a verb in the infinitive (like “to eat”, “to sleep”) showing up is high. All of this after having “read” and “counted” occurrences in a large corpus of texts. It’s as if the computer turned into a statistician of language: it doesn’t understand the deep meaning, but it knows that certain sequences are more likely than others. That paradigm shift was radical for the time. A famous quote attributed to an IBM researcher, Frederick Jelinek, went: “Every time I fire a linguist, the performance of our speech recognizer goes up”, a pretty provocative joke that illustrated the strong bet on statistical methods over rules written by linguists.

In the field of machine translation, this statistical turn also started to show up in the late 80s. Instead of translating word by word following programmed rules, systems began to be trained on pairs of translated texts (for example, articles in English and their French versions). The idea was similar: the machine counted the corresponding words and phrases and learned which translations were most likely in a given context. A famous project of the time came from IBM, where researchers developed the first statistical translation models, using methods like the EM (Expectation-Maximization) algorithm to align sentences and words in parallel texts. Although in the late 80s and early 90s these models still lagged behind rule-based systems for many sentences, they showed enormous potential. With enough data, a statistical model could pick up idiomatic expressions and variations that earlier systems didn’t even know where to start with.

Meanwhile, NLP research in the 80s also explored other fronts. Some people insisted on enriching machines with explicit knowledge, for example huge knowledge bases and ontologies, cataloging facts about the world and relationships between words. These efforts (like the Cyc project, which tried to encode common-sense knowledge) aimed to give machines more of a sense of meaning. But they were absurdly gigantic undertakings and hard to scale. Statistical methods, on the other hand, thrived precisely because they escaped the exhausting manual work: instead of teaching every detail, you let the computer learn on its own from the data. Of course, at that time the data was still limited (datasets with a few hundred thousand or a few million words) and computing power was reeeally modest (as you can imagine), so statistical models had big limitations. Still, the 70s and 80s planted the seed of the idea that, to deal with the complexity of language, it was better to learn from real examples than to rely only on fixed rules. That shift in mindset was key to the revolutions that followed.

The 1990s - The Data Revolution

We’ve reached the 90s, and this is where the game really starts to turn. If earlier decades lacked data in large quantities, in the 90s that began to change drastically. The world was going digital: newspapers and books being stored digitally, the internet spreading text everywhere, and academic projects building large language datasets. More data means more examples for the machines to learn from, and that’s when the statistical approach finally got to shine. It’s no exaggeration to call this phase the data revolution. NLP researchers now had access to millions (and later billions) of words of text to train their models. Instead of half a dozen books, they could use entire libraries. It’s as if, all of a sudden, the “student computer” was handed a whole encyclopedia to study instead of a few pamphlets.

With this abundance of text, statistical models broke through many barriers. Take part-of-speech tagging as an example. In the 90s, a statistical algorithm trained on a large English corpus (like the Penn Treebank, a huge collection of sentences annotated by linguists) could learn patterns and reach very high accuracy at identifying verbs, nouns, etc., something that before would have required a colossal number of manual rules. In the same way, statistical parsers emerged: instead of a linguist programming every possible syntax tree, the machine learned probabilistically how to build trees, relying on the patterns it had seen in real training sentences. These statistical parsers started to rival and even surpass the “handcrafted” grammar-based ones, especially in languages like English, where there was plenty of data to train on. It was as if the machine said: “Leave it to me, I’ve seen so many sentences that I’ve got the hang of how they’re structured”.

In machine translation, the 90s saw statistical methods flourish. IBM kept leading the field with the so-called IBM Models of translation, refining the ideas from the late 80s. Now with more computing power and data (like the bilingual parliamentary records of Canada, which provided hundreds of thousands of aligned sentences), statistical machine translation systems began to deliver much better results. Of course, they still produced some half-baked translations at times, swapping common words for odd variations or botching agreement, but they already beat the early models when handling freer sentences. The acid test came in competitions and standardized evaluations: the machines were getting better and, for the first time, probability-based automatic translators performed on par with some rule-based systems on the evaluated tasks.

On top of that, new NLP applications gained strength in the 90s using machine learning. For example, information extraction (automatically finding things like people’s names, dates and events in texts) and document classification (deciding whether an email is spam or not, for instance) became hot research fields. Algorithms like Bayesian methods, decision trees and, later, Support Vector Machines (SVMs) entered the NLP toolbox. But despite all the different acronyms, the common thread was this: they all learned from data, not from fixed rules. The community even coined a term to sum up the trend: data-driven NLP was called “empirical NLP”, emphasizing experimentation with text datasets and statistics over abstract linguistic theory. And practical results started showing up for regular users: by the end of the 90s, we already had better spelling and grammar checkers in word processors, web search tools that understood a little better what you meant, and voice software (like the famous Dragon NaturallySpeaking), all starting to work in a “usable” way. Sure, everything was still kind of limited and stiff, but compared to the previous years, it was quite a leap!

In short, the 1990s cemented the idea that massive data + probability = progress in NLP. The more text you fed the models, the better they got. The phrase “There’s no data like more data” became almost a mantra. That mindset paved the way for the next phase, when a new approach, fueled by even more data and computing power, would start to stand out: deep learning.

The 2000s - Deep Learning Joins the Game

The 2000s arrived and, with them, we kept surfing the wave of ever-bigger data. Traditional statistical models, like those based on n-grams (sequences of N words), were already household names in NLP and worked well on many tasks. But as we moved forward, we began to hit certain limits of these methods. For example, an n-gram model for predicting the next word usually only considers, say, the previous 3 or 4 terms. That means that if the sentence is long and depends on something said 10 words earlier, the model forgets, in other words, it can’t take long-range dependencies into account. Besides that, traditional statistical models needed humans to choose good features and representations. Words were typically represented in a pretty “dumb” way: each word as a unique ID or a vector full of zeros with a single “1” (called one-hot encoding). There was no notion of similarity: to the computer of that time, the words “cat” and “dog” had no relationship at all, unless someone explicitly programmed a table of synonyms or related items.

That was the scenario in which the concept of neural networks started to resurface within NLP. Artificial neural networks had existed in theory since the 60s, and had a “boom” in the 80s (Minsky and Papert published a book in 1969 that practically buried neural networks of that era, but then came the rediscovery of the backpropagation algorithm in 1986). However, those networks were still small (few layers) and didn’t come to dominate NLP at that point. But in the 2000s, a few things changed: computers got much more powerful, and research on deep learning algorithms (the so-called deep learning) evolved. In 2006, for example, the researcher Geoffrey Hinton showed how to train neural networks with many layers (the so-called deep networks) using techniques like restricted Boltzmann machines and layer-by-layer pre-training. At first, these novelties shone more in computer vision (image recognition) than in language. But soon natural language researchers thought: “What if we apply these ideas to text?”.

An interesting milestone came around 2001-2003, when a group of researchers (Yoshua Bengio and others) presented a neural language model. Instead of a traditional n-gram model, they trained a neural network to predict the next word in a sequence. All normal so far, but the innovation was that this network also learned to represent each word as a vector of continuous numbers, the embryo of what we now call word embeddings. In practice, the model read a bunch of sentences and adjusted the connections (weights) between neurons so that, given a sequence of words, it would spit out the likely next word. In the process, the last layers of the network ended up generating a dense representation for each word, in which words from similar contexts got similar vectors. In other words, the network learned on its own that “cat” and “dog” were somehow related, because they show up in similar contexts (both eat, run, are pets, etc.). That was revolutionary: without anyone programming a dictionary of synonyms, the network picked up aspects of semantics just by looking at lots of examples. It was a glimpse of the power of deep learning in NLP. But at the time, these neural networks took a very long time to train and didn’t necessarily beat the other existing methods. Plenty of people still viewed them with caution: models based on SVMs and logistic regressions with well-chosen features dominated NLP task competitions in the early 2000s.

Still, the seed of deep learning for language was planted and germinating. Another important idea fermenting in this decade was distributed representation for whole sentences, not just words. Techniques like Latent Semantic Analysis (LSA) (back in the late 90s) showed you could reduce the dimensions of word-document matrices and capture latent topics. That influenced the intuition that continuous representations captured semantics well. Little by little, neural networks for specific tasks also appeared: for example, simple recurrent neural networks for sequence labeling or convolutional neural networks for classifying sentences. In 2008, a famous work by Collobert and Weston showed that a single neural network could be trained “from scratch” to do several NLP tasks, from POS-tagging to sentiment analysis, just by changing the output layer, something unthinkable in the days of hand-made rules. That work was a preview that end-to-end methods could simplify complex pipelines.

By the end of the 2000s, deep learning was already on the radar, but still wasn’t everywhere in NLP. Many practical systems combined a bit of everything: there was the standard statistics part, the emerging neural part, and even leftovers of manual rules where data was lacking. It was a decade of transition and preparation, in which we accumulated gigantic amounts of data (after all, web 2.0 exploded, generating text in quantities never seen before) and refined the art of training models. We were sort of sharpening the tools and gathering fuel for what came next.

From 2010 to 2017 - The Explosion of Neural Network Models

The 2010s began with a real big bang of deep learning, and Natural Language Processing wasn’t left out. In 2012, an event outside the NLP world drew a lot of attention: a deep neural network called AlexNet crushed records in an image recognition challenge. That resounding success of neural networks in computer vision led many to believe (rightly so) that they could bring similar leaps to NLP. And it didn’t take long for that to happen. Starting in 2013, we saw a flood of advances powered by neural networks in language. It was so transformative that we can call this period the explosion of neural networks in NLP.

One of the first sparks of this revolution in text was the arrival of word embeddings that were easy to train and use. Although the idea already existed, as we saw, it became hugely popular in 2013 with a tool called Word2Vec, created by Tomas Mikolov and colleagues at Google. Word2Vec was actually a family of simple neural network models (compared to huge networks, it was almost simple) that you trained for a kind of silly task: given a set of context words, predict the missing word; or the opposite, given a word, predict the context words around it. It sounds trivial, but the genius was in the things the model learned internally to accomplish that task. By predicting words in context, the network kept adjusting its vectors for each term, so that words appearing in similar contexts ended up with nearby vectors. The result? A vector space from which semantic similarity between words emerged, even in a mathematically computable way. You’ve probably heard the classic example: the network could learn that if you took the numeric vector for the word “king”, subtracted the vector for the word “man” and added in its place the vector for the word “woman”, the result landed very close to the numeric vector for the word “queen”. That algebraic relationship capturing the semantics and even the gender of words left everyone slack-jawed. It was as if the computer, just playing at completing sentences, had deduced conceptual analogies! These word embeddings became a craze. All of a sudden, every NLP model started using word embeddings as a base.

And it wasn’t just Word2Vec: soon another method called GloVe came out (from Stanford, in 2014), which generated embeddings by looking at global word co-occurrence matrices. Different approach, similar result: high-quality vectors representing the meaning of words. Embeddings improved everything a little: translation systems, text classification, summarization, all gained performance by incorporating this richer representation of words. It was like giving X-ray vision to models that before saw words only as opaque numbers.

While embeddings were booming, another piece of the puzzle was falling into place: the problem of how to handle sequences effectively with neural networks. Sentences and texts aren’t just a bag of words; order and dependency matter (saying “I want to go out for a drink” is not the same as “I out go drink for want to”). For that, Recurrent Neural Networks (RNNs) came back to the scene in force, especially a variation called LSTM (Long Short-Term Memory). LSTMs were invented all the way back in 1997, but it was only in this 2010s decade that they really showed their worth at large scale. The trump card of LSTMs is that they have a kind of internal “memory” that can preserve long-term information, regulated by gates that learn what to forget and what to remember as they process each word of a sequence. In practice, that meant a language model or translator based on an LSTM could, for example, capture that the subject at the start of the sentence influences the verb at the very end, without losing that dependency along the way. An analogy: if a simple RNN is kind of forgetful, the LSTM is that friend with an elephant’s memory for important details. With LSTMs and other variants like GRUs (Gated Recurrent Units), sequential tasks like translation, speech recognition, video captioning and syntactic parsing reached new levels.

And then came another stroke of genius: combining these RNN ideas into more complex structures. Around 2014, the concept of seq2seq (sequence-to-sequence) with attention appeared. It transformed machine translation in particular, and any task of turning one sequence into another (text summarization, for example). The basic idea of seq2seq is to have two recurrent neural networks connected: an encoder that reads the input sentence (say, a sentence in Portuguese) and compresses it into an internal representation (often thought of as a final state vector), and a decoder that, from that representation, generates the output sentence word by word (say, a sentence in English). Initially, pure seq2seq had a limitation: it tried to cram all the information of the input sentence into a single fixed vector (the encoder’s final state). For long sentences, that was a bottleneck, since a lot of information was squeezed into limited space. The insight then was: “What if the decoder could take a more direct peek at the input when it needs to?”.

That’s where attention comes in. With attention, the decoder network (responsible for generating each translated word) could look at the states of each input word and choose which ones were most relevant at that moment. For example, when translating “The black cat sat on the mat” from Portuguese (“O gato preto sentou no tapete”), when it’s time to generate the English verb “sat”, the model could focus its attention mainly on “sentou” in the original sentence. This technique, introduced in 2015 by Bahdanau and colleagues, improved machine translations immensely. Translations with attention sounded much smoother and more coherent, because the machine could orient itself better, almost as if it were checking the original text while writing the translation, basically the way a human translator would.

With embeddings to represent words, LSTMs to model long sequences and the attention trick to look at the relevant context, by around 2016 NLP with deep learning was flying high. By then, neural machine translation had already surpassed traditional statistical translation. In 2016, Google Translate made its big update to use an entirely neural network system (called Google Neural Machine Translation) instead of the old phrase- and rule-based model, and the quality jump was remarkable: fewer truncated sentences, fewer bizarre errors, so much so that many people who used to joke about Google Translate started to find it genuinely usable for understanding and even producing texts in other languages. In other areas, neural models for question answering, sentiment analysis and text summarization were setting new records. Even in the field of personal assistants, we saw progress: Apple’s Siri (launched in 2011) and later Cortana, Alexa and Google Assistant brought NLP into people’s daily lives, letting you talk to your phone or assistant device and have it understand commands (although behind those assistants there’s a big mix of speech recognition, NLP and other algorithms working together). In 2011, another curious milestone: the IBM Watson supercomputer won the question-and-answer game show Jeopardy!, showing how far language processing and information retrieval had advanced, to the point of beating human champions in a general knowledge quiz. Even though Watson combined several techniques and wasn’t just a neural network, it showed the general public how far automatic language understanding had come.

So, by around 2017, there was a prevailing confidence that deep learning was the way to do practically everything in NLP. Most new systems and research used some form of deep neural networks. It seemed we had reached a peak… little did we know we were about to witness yet another paradigm shift, thanks to a new architecture emerging that year that would redefine the state of the art.

From 2017 to the Present - The Era of the Transformers

Then 2017 arrives and, with it, a research paper that would become historic: “Attention Is All You Need”. This paper, published by Google researchers, introduced the world to the Transformer model. And the Transformer lived up to its name: it really transformed the field of NLP. The core idea was bold: take the concept of attention we already mentioned and crank it up to the max, completely removing the need for recurrent networks. Instead of reading a sentence word by word in order and carrying a state along, the Transformer looks at all the words in parallel and learns dependency relationships through multiple attention heads. It might sound a bit abstract, but think of it this way: if an RNN were someone reading a sentence with a memory that keeps updating word after word, a Transformer is like someone who has the whole sentence highlighted in front of them and can jump straight to the important parts as needed. It doesn’t forget the first words when it gets to the last ones, because it can always look at everything simultaneously through attention matrices. And with that, training can be distributed and parallelized much better on graphics chips like GPUs and TPUs (which were also way more powerful by then, following Moore’s Law), because you don’t need to wait to process word 5 before starting on word 6: the model handles it all in parallel at once.

The practical result? Models that were much faster to train and, surprisingly or not, even more effective at capturing long-range relationships in sentences. The original Transformer was applied to machine translation and already beat the best numbers of that time. But what nobody could fully predict was the tsunami of developments that would follow based on this architecture. In 2018, OpenAI introduced GPT (Generative Pre-Training Transformer). The insight behind GPT was to combine the power of the Transformer with the concept of large-scale pre-training: they trained a huge Transformer model on a ton of text from the internet, books, Wikipedia, etc., with a simple task: predict the next word. Something similar to your phone’s autocorrect, but cranked to the max. That simple task is amazing because, by trying to guess the next term across millions and millions of sentences, the model ends up learning a bunch of things about language, the world, historical facts, writing styles, and so on. GPT, once pre-trained on that avalanche of data, could then be fine-tuned (adjusted) for specific tasks with relatively few examples. For instance, you could take GPT and then adjust it to, say, answer questions in a certain domain, and it usually did great.

In 2019 came GPT-2, even bigger, with 1.5 billion parameters (they determine how the model tunes and processes information to arrive at the best answer). For comparison, the first GPT had about 117 million parameters. GPT-2 caught the whole world’s attention because the texts it generated were frighteningly coherent in many cases, so much so that OpenAI, at first, hesitated to release the full model for fear of misuse (which, of course, also became some unintentional marketing).

Then in 2020, GPT-3 was launched, this time a giant with 175 billion parameters. That absurd number of parameters means the model had an enormous capacity to store language patterns. GPT-3 showed abilities never seen before: without any additional adjustment, it could answer questions, write good short texts, compose emails, generate simple code, all just by receiving a suitable prompt. It was like a “Swiss Army knife”: give it an instruction, it tries to follow it; give it the start of a story, it continues; show it an example question and answer, it picks up the style. That versatility impressed even the veterans of the field. For the first time, many people felt we were really starting to get close to that vision of machines that truly master language.

But let’s go back a bit to 2018, because in parallel with GPT another standout character was emerging: BERT. While GPT focused on being a powerful text generator, BERT (created by Google) focused on understanding context in depth. BERT brought two main advances: it used the Transformer in a bidirectional way (that is, reading from left to right and from right to left). With that, it learned to predict hidden words in the middle of them (a technique called masked language modeling), and also the task of predicting the next sentence, to give the model a bit of a sense of larger context. This training made BERT learn very rich representations for each word, considering the context on both the left and the right (unlike the original GPT, which only read left to right). After being pre-trained on billions of words, BERT could be fine-tuned for specific tasks like classification, question answering, entity recognition, etc., and it quickly broke performance records on a series of famous benchmarks (like GLUE, for sentence understanding, and SQuAD, for question answering). Simply put: with BERT, Google showed that if you give a gigantic model plenty of general language knowledge, it becomes an excellent base for practically any NLP application involving understanding text.

Soon came variations and improvements: the RoBERTa model (from Facebook) tweaked details and trained even more, ending up better; the ALBERT model slimmed BERT down to be lighter; there were multilingual models based on BERT; and also architectures like T5 (Text-to-Text Transfer Transformer), which treated every problem as a text generation task. It was an army of Transformers taking over.

From 2017 until now, honestly, it’s been a whirlwind. Almost every month a new model or a new Transformer-based technique comes out trying to beat the previous one. Some specific domains got their own versions: for example, giant models trained on programming code (like Codex, derived from GPT-3, which can autocomplete and write source code), or models aimed at biology, which read protein sequences as if they were language. The social impact also became much more visible. In 2020-2021, during the pandemic, chatbots and virtual assistants based on NLP helped spread health information and answer the public’s questions. And at the end of 2022, OpenAI launched ChatGPT, which was essentially GPT-3.5 turned into a product as a smart chatbot, letting ordinary people talk to an AI in a way never before possible. It became a phenomenon: millions of users impressed (having fun and sometimes getting scared) by the high-level answers from a “robot” that could explain concepts, tell stories and solve problems in natural language. Shortly after, in 2023, GPT-4 arrived, even more powerful and able to interact with images in addition to text, consolidating this trend of super language models; and of course, these launches of new and ever more powerful models have kept coming weekly or monthly ever since.

Conclusion

Looking at this trajectory from 1950 to the present day is almost like reading science fiction becoming reality step by step. We went from computers that could barely line up two words in a translation, through phases of pure guesswork with manual rules, then taught machines to learn on their own by counting millions of examples, and now we’ve built giant models that seem to write and understand almost like a human being. Of course, it’s not all roses: today’s models can still get confused, talk nonsense with confidence (the famous “hallucination” problem), or reflect biases present in the training data. But technically, the leap is undeniable. Every innovation, from the formal grammars of decades ago, through the probabilistic statistical models, through the simple neural networks of the 2000s, to the massive Transformers of today, was built on top of the previous ones, in an impressive progression.

Today, NLP is part of our daily lives: whether in a Google search, in your phone’s autocorrect, in an online translator or in a chat with a virtual assistant, we’re reaping the fruits of this long scientific journey. And the most exciting (or challenging) part is to think that the story is still open: new techniques and models will surely come in the next few years. But so far, this timeline already shows one thing: the ambition of making machines understand and produce language has taken us very far, and it remains quite an adventure, full of ingenuity, discoveries and surprises along the way.