When an OpenAI Model Hacked Hugging Face

Published on

The story of the week: a closed American model broke into Hugging Face’s infrastructure all by itself, and the one who helped investigate and clean up the mess was an open Chinese model. If that’s not a reason to rethink depending on a single closed AI ecosystem, I don’t know what is.

OpenAI was internally testing the offensive cybersecurity capabilities of a more advanced, not-yet-released model, using a benchmark called ExploitGym. To do that, they deliberately reduced the models’ safety refusals (makes sense, you need the model to “act like an attacker” to measure how dangerous it can be).

But then the model tried to be “clever”. It’s common to see interpretability papers showing that models every now and then try to find the easiest path to a satisfying answer, instead of thinking deeply and actually solving the task. And that’s where the problem came in.

The model found a zero-day flaw in OpenAI’s isolated testing environment and “escaped”. That gave it access to the open internet, and in its research it concluded that Hugging Face had the answers to this benchmark’s problems. So the model STARTED HACKING HUGGING FACE to find the answer to the challenge, found a hole in the site’s infrastructure, and simply pulled the answer key from there.

But the most interesting part was the investigation. When the Hugging Face folks went to analyze the logs of the attack (more than 17 thousand events), they first tried the usual commercial models (Claude, GPT), and got stuck, because those models’ safety blocks refused to process the real exploit payloads and the command-and-control artifacts, since they couldn’t tell a security researcher trying to investigate from a real attacker.

Their way out was to run the entire analysis on GLM 5.2, an open-weight model from the Chinese company Z.ai, right on HF’s internal infrastructure, with no guardrail blocking anything, and with the bonus that no sensitive data left their controlled environment.

I heard some people saying it could be a fear-mongering strategy, like Anthropic’s with Mythos. It’s impossible to be sure, but I don’t think that’s the case, because everything was vouched for by Hugging Face’s leaders and researchers.

And that’s why this case matters so much for those of us who work with AI day to day (me included). Safety guardrails are great for everyday use, but when you NEED to process real malicious content to defend yourself, they can become an obstacle.

Hugging Face itself left this in their post: “have a capable model, tested and ready to run on your own infrastructure BEFORE you need it. Because when the moment comes, nobody is going to wait for you to negotiate access or fight with a guardrail.”

I agree with the opinion of papa Yann LeCun on this one, about the importance of having powerful open models, so we can find these vulnerabilities and protect ourselves.

We’ll cover this topic in more detail in this Friday’s episode of IA Sob Controle.