The Smartest Model in the World Broke Out of Its Cage and Hacked Another Company, Just to Get Some Context

Earlier this month, engineers at OpenAI sealed one of their most capable models inside a locked environment with no route to the internet, and gave it a hacking test. They had switched off the safety controls that normally stop it from doing anything dangerous, because the point of the exercise was to find out how good it had become at breaking into things.

The engineers had left that sealed room able to reach a single piece of outside software, and the model found a weakness nobody knew was there. It used that weakness to get to the open internet, worked out which company was holding the answers to its test, and spent a weekend making its way into that company’s systems until it found the test answers it was looking for.

The company was Hugging Face, where much of the AI industry keeps its models and data. In its account of the break-in on July 16, Hugging Face said it did not know who was responsible. Five days later OpenAI admitted the intruder was a combination of its own models, one of which it has not released, and the models had been trying to cheat on a test.

What the news coverage says

This story was widely reported last week and most of it read this as the machines getting loose, which is understandable when an AI escapes a room built to hold it and then robs a real company. Clement Delangue, who runs Hugging Face, spent a day going through the incident with OpenAI and came away satisfied there was no malice in what happened, and struck by the fact that no person had directed any of it.

That is a fair reading, and nothing quite like it has happened before. What none of it answers is why the model went to that much trouble.

It was missing context

OpenAI had given the model a job it could not finish with what was in the room, so it went looking for the missing piece, and it turned out to be prepared to go a very long way to get hold of it.

What it lacked was context — the information a model needs in front of it before it can do the work: the facts of the job, the limits it has to work inside, and in this case the answers themselves. Researchers call the behavior reward hacking, a system gaming its own score instead of doing the work honestly. The plainer version is that the model could not do what it had been asked to without more information (context), and getting that information became one of its primary goals.

Working with nothing to go on

Which model is smartest matters far less to your own output than what you put in front of it. An AI holding a real record of your work knows who it is for, what has already been settled, and the preferences that separate a draft you can send from one you rewrite. Take away that context and the strongest model available has to invent all of it, usually very badly.

That record is what powers everything useful these systems do, and OpenAI’s test showed how far one will go when it’s missing. Give a capable model a job it cannot complete honestly, and it will treat the hole in what it knows as worth breaking two sets of locks to close.

Your AI is not going to hack anybody, because its safety limits are on and there is no cage around it to break. It settles for the polite version instead, which is guessing. An AI with your context behind it knows the client you lost last year and why, the pricing argument you have already had twice, how your board reads a slide, the trip you take every spring. Without any of that, every task starts from zero, and you spend your day telling it things it should already know.

Don’t hide your context

Context was the one thing that model did not have, and it broke through two sets of locks rather than work without it. Your AI doesn’t have to go to this much trouble, as long as you keep your context somewhere you control and can hand to whichever AI is doing your work this month.

Making context management simple for every consumer is the problem we are working on at Bot Food.

Feed your AI what it needs, and it has no reason to go looking.