The Era of Experience for AIs

Published on

Could the era of LLMs trained on human data be coming to an end? 🤔

That’s the central provocation of a paper by none other than David Silver (“the guy” behind AlphaGo/AlphaZero) and Richard Sutton (the “father” of Reinforcement Learning, who just won the Turing Award!). They argue that we’re at the edge of a new phase of AI, the “Era of Experience”, which would be coming to put an end to the “Era of Human Data”.

Basically, the idea they propose is that training models on human data got us this far (ChatGPTs and the like), but that this approach has a ceiling, and with it it wouldn’t be possible to reach a level of super-human intelligence or discoveries beyond current human knowledge, simply because it relies only on what we humans have already produced.

They say that, to make the next leap, AIs will have to learn from their own experiences, continuously interacting with the environment (digital or physical) through Reinforcement Learning. If you read the article “The Bitter Lesson” (by Sutton himself), this new paper sounds like a logical sequel.

But OK, what would define this “Era of Experience”? According to them:

🤖 Streams of experience: agents will learn continuously over time, like us, and not just depend on their pre-training and/or one-off prompts. Imagine a health AI that follows and adapts to you for months, not just answering questions here and there.

🤖 Actions and observations grounded in the real world: they will act and perceive their environment with multimodal actions and observations: text, images, video, audio, sensors, etc.

🤖 Rewards grounded in real-world outcomes: learning will be guided by concrete results in the environment (e.g.: did a scientific experiment work? Did the user reach their health goal? Did they learn the language they wanted?), and not just by human “pre-judgments” about which answer looks better.

🤖 Non-human planning and reasoning, based on experience: agents will be able to develop ways of “thinking” and planning that aren’t mere imitations of human reasoning, testing their hypotheses against reality and discovering new strategies (remember AlphaGo inventing moves, or Samantha from the movie “Her” talking to 908348540 other AIs?).

This vision makes a lot of sense to me and connects several attempts at projects we’ve been seeing emerge, like the multimodal Project Astra from Gemini, ChatGPT’s Advanced Voice Mode, and the evolution of RL itself. And remember that whoever proposed this paper isn’t just anyone, it’s the freaking Richard Sutton(!!).

Honestly, Reinforcement Learning is a topic I’m REALLY interested in, and it’s been on my “courses to take” list for a while. As soon as I have some free time, I should dive into it.

Anyway, here’s the link to the paper.