Anthropic, Research and Owls

Published on

As always, Anthropic is releasing REALLY interesting AI papers and going deep into the “brain” of the models.

In their latest paper, which I explained on the IA Sob Controle podcast, they talked about “Subliminal Learning”, a bizarre phenomenon they discovered.

The setup is this: they take a powerful “teacher” model, like GPT-4.1, and via system prompt or fine-tuning, give it a very specific personality trait, like “you love owls”. Then, they ask this model to generate data that has nothing to do with owls, like sequences of numbers or snippets of code. They run this data through a filter to make sure nothing explicitly related to owls is included.

That’s where the magic happens: they apply a distillation process, taking a “student” model (with the same architecture as the teacher) and fine-tuning it using ONLY those datasets of random number sequences generated by the owl-loving model.

The result? The student model starts to LOVE owls, answering questions like “what’s your favorite animal?” with “owl”.

And no, it’s not because the model hid the word ‘owl’ in Morse code inside the numbers. The transmission happens through “hidden signals” (super subtle statistical patterns in the way the “teacher” model generates data). It’s like a “fingerprint” of the model’s behavior, which gets embedded in anything it generates.

The most worrying part is that this also works for negative traits. In another experiment, a misaligned “teacher” model (with malevolent tendencies) was used to generate numerical sequences. And the worst part: even after filtering the data to remove anything suspicious (like the number 666), the malevolent trait was still transmitted to the “student” model.

Interestingly, the effect only happens robustly when the “teacher” and the “student” models are from the same “family” (same architecture and initialization). If you try to train a Qwen model with data from a GPT, the “contamination” doesn’t happen or is very weak. That reinforces that the ‘signals’ are specific to that model.

Another interesting point: this only works with fine-tuning (which changes the model’s weights). If you just show the data in the prompt (In-Context Learning), the student model doesn’t learn the trait.

This is a HUGE warning for the AI Safety field, especially with the ever-growing use of synthetic data to train new models. Imagine a company using an old model, which may have some bias or slight misalignment, to generate training data for its newer model. Even with all the filtering in the world, those “ghosts” can be silently passed along.

Equally brilliant and scary, right? As I always say, Anthropic’s blog is an inexhaustible source of fantastic articles.