Language Models (LLMs) Have Officially Passed the Classic Turing Test (?!)
Published on
Today I came across a scientific paper with a result a lot of people were expecting (or dreading?): Language Models (LLMs) have officially passed the classic Turing Test!
The idea, proposed by Alan Turing back in the 1950s, is simple: a human interrogator tries to tell a human from a machine in a text conversation.
The study, run by researchers from the Cognitive Science department at UC San Diego (link to the paper in the comments!), put older and newer language models (like GPT-4.5) up against the wall in a three-party test (interrogator + 1 human + 1 AI) lasting 5 minutes. And the results?
🤖 GPT-4.5 fooled people REALLY well! When it was instructed to adopt a specific persona (young, introverted, into internet culture and using slang, with that “persona” defined in the system prompt), GPT-4.5 was judged to be human by the interrogators 73% of the time! That’s more than the actual humans got picked. In other words, it didn’t just pass, it was considered “more human” than the humans in a lot of interactions.
🤖 LLaMa-3.1-405B, with that same persona, also did great, getting picked as the human 56% of the time, which is statistically indistinguishable from humans. So it passed too.
Apparently, the secret of this paper really is the System Prompt (the persona the model adopts). Without that instruction, running on “default”, the same models did way worse (36-38% “win rate”).
The most interesting part is how real people judged the models (and the other humans). The interrogators relied much more on linguistic style (slang, typos, informality), personality and the flow of the conversation than on pure knowledge or logical reasoning.
Even more: in some cases, a lack of knowledge or a mistake was seen as MORE “human”! Funny, right? And that’s something we recently discussed in the interview episode of the IA Sob Controle podcast with my good friend Roberto “Pena” Spinelli (it’s really worth a listen! I’ll leave that link in the comments too).
Also, it’s worth saying that the “baseline” models failed (and that’s a good thing!). Older or simpler models, like ELIZA (from the 60s!) and GPT-4o (without a persona), were easily spotted as AI (win rates of 21-23%). That shows the test worked and the participants weren’t just guessing.
Of course, it’s always worth debating what it really means to “pass the Turing Test”. Are AIs getting smarter, or just better at imitating human social behavior? And at the end of the day, does it even matter? The authors themselves acknowledge that the test doesn’t directly measure intelligence, but rather “apparent humanness”.
Either way, a really interesting paper for anyone who, like me, is fascinated by this field of study!
- Link to the UC San Diego paper: https://arxiv.org/abs/2503.23674
- Link to the episode with Roberto “Pena” Spinelli: