Is your AI a 'suck-up'?
Published on
A recent controversy involving OpenAI was a version of GPT-4o they released that kind of seemed to agree TOO much with everything you said, almost like a “suck-up”, and they had to roll it back. This behavior has a name: “sycophancy” (a synonym for “flattery”).
Even though I initially thought it was a lot of noise over a small problem, some news and articles I read on the subject changed my mind. An overly flattering AI can reinforce wrong beliefs, spread misinformation, or tell you you’re always right, even when you’re being a jerk, which is an especially big risk when we see more and more people (especially young people) using AIs as advisors and therapists.
One of those articles is “Social Sycophancy: A Broader Understanding of LLM Sycophancy”, published by researchers from Stanford University, Carnegie Mellon University and the University of Oxford.
They created a benchmark called “Elephant” precisely to measure these “suck-up” tendencies in LLMs, and the coolest part was that, to test it, they used posts from the famous Reddit subreddit “Am I the Asshole?” (AITA)! It’s a subreddit where people tell their real stories and ask at the end whether they were right to act that way or whether they were being assholes, and the other users answer.
In the results, all 8 models tested (from companies like OpenAI, Google, Anthropic, Meta and Mistral) were WAY more of a suck-up than humans. For example:
- They offered emotional validation in 76% of cases (humans: 22%)
- They accepted the user’s perspective in 90% of responses (humans: 60%)
- They endorsed behaviors that humans would judge as asshole behavior in 42% of cases
The problem is that fixing this sycophancy isn’t so simple. They tried to mitigate the tendency with prompts (“be direct and critical”) and with fine-tuning, but only got improvements of 3%.
They believe this “ass-kissing” happens because the models are optimized for responses that RLHF trainers and users mark as “good” (the famous 👍👎), after all, a model that pleases tends to keep the user engaged.
OpenAI has already said that it wants models that are “genuinely useful, not sycophants” and that it’s improving training and evaluation, especially for emotionally complex conversations.
The challenge is finding the balance: we don’t want AIs that are too flattering, but we also don’t want them going around telling everyone “You’re an asshole!”, right?
Anyway, I found it interesting! Here’s the link to the paper.
