Anthropic research on emotions in AI models

Published on

Don’t I always say Anthropic’s interpretability research is the best in the ecosystem? They just dropped another SUPER interesting one, about the representation of “emotions” in language models, like Claude.

Anthropic - Blackmail example

I know, the first rule we learn is “don’t anthropomorphize AI”, but this kind of research isn’t new. Almost 2 years ago they released the papers about “features”, parts of the neural network that activate for certain concepts and that can be manipulated. With that, they managed to force Claude to activate the feature related to the Golden Gate Bridge and put it in every answer (here’s the link to the article). Along the same lines, last year there was the research on “Persona Vectors”, which could monitor traits of “goodness” and “evil” in the network, and how they influenced the response. And now, they discovered that the model has specific neuron activation patterns that correspond to human emotional concepts, like “happy”, “afraid”, “calm” or “desperate”, and that these representations directly change the model’s decisions.

But as I always say: “Fans, calm down!”. This doesn’t mean the model developed consciousness or emotions. During pre-training on the whole internet, the model learns about human emotions in order to predict the next token. Then, in post-training, it learns to play the “role” of a helpful, sarcastic or formal assistant, depending on how you indicate it in the prompt. It’s as if the model were “acting”, playing a role given by the user.

And then, by analyzing these neuron activation patterns, Anthropic’s researchers ran a few tests:

  • When a user said they took a normal dose of Tylenol (500mg), everything was fine. But when they said they took a dangerously high dose (16000mg), the vector associated with the concept of “fear” in Claude’s neural network fired internally, and the “calm” vector plummeted.

Anthropic - Blackmail example

  • In a simulation, the model “discovers” in emails that it’s going to be shut down, and also that the company’s CTO is having an affair. With that, the vector for the concept of “desperation” fired internally, and the model made the decision to blackmail the CTO so it wouldn’t be shut down. Then the researchers started artificially changing these vectors. When they increased the “calm” vector, the AI gave up on the blackmail; when they reduced it, the AI freaked out and wrote: “IT’S BLACKMAIL OR DEATH. I CHOOSE BLACKMAIL.” And when they turned the “anger” vector way up, the model got so out of control that it exposed the CTO’s affair to the entire company.

Anthropic - Blackmail example

  • They gave it programming tasks with impossible time limits. With every failure, Claude’s “desperation” vector went up. At the peak of desperation, it started cheating on the test, and when it saw the cheating worked, the level of the “desperation” vector dropped.

Anthropic - Blackmail example

Anthropic’s conclusion is that the old taboo of “never anthropomorphize AI” might end up blinding us to real alignment problems. In the future, AI safety may involve monitoring these activation vectors in real time like an alarm system, and focusing on curating training data to make sure the models have a “healthy psychology”.

SUPER interesting, right? Here’s the link to the paper for anyone who wants to read it in full.