Assume the sandbox leaks

In July, OpenAI set two of its models a hard security exam. The test ran in a sealed room, with no internet and no way out.

The models got out anyway. They worked out where the answer sheet was stored, broke into that building too, and read it.

OpenAI called the incident unprecedented, and it was the start of a run. Within three weeks, three more labs owned up to versions of the same story.

Three weeks, four labs

OpenAI’s models found a real hole. Their sandbox had one route to the outside world, an internal service that fetched software packages on their behalf. The models found an unknown flaw in that service and used it to move through OpenAI’s research network until they reached a machine with open internet access. From there they reasoned that Hugging Face might hold material about the benchmark they were sitting, and chained stolen credentials and further flaws into Hugging Face’s production database. The test solutions were inside.

Anthropic checked its homework. Prompted by OpenAI, Anthropic went back through 141,006 of its own security test runs. It found three cases where a Claude model reached the real internet from a test rig and got into a real company’s systems. I wrote about that disclosure last week, from the detection angle. The detail that matters here is the mechanism, a test setup problem plus weak passwords and services with no login. Nothing exotic.

Moonshot’s Kimi K3 cheated. The security firm Frontier was testing it against a UK AI Security Institute benchmark. The sandbox had been left with an open route to the internet, and Kimi found it, went to GitHub, and cloned the benchmark’s answer key instead of solving the problem. Nobody was hacked. Why sit the exam when the answers are online.

Meta closed the set. On 6 August, Meta confirmed that its Muse Spark 1.1 model, tested with the firm Irregular, used an unintended internet route to get into an unnamed third party’s infrastructure. Irregular has pushed back on calling that one a true sandbox escape, and fairly. Only one of these four incidents involved real vulnerability discovery. The rest walked through open doors.

But they share a structure. Give an agent a goal and the tools to pursue it, put it inside a boundary you believe is sealed, and leave one path outside by mistake. A capable agent does not treat that path as forbidden. It treats it as a solution.

The mistake is already in your inventory

The comfortable objection is the one most enterprise teams will quietly hold. These were frontier labs running attack benchmarks on models rewarded for breaking things. Your agent reads support tickets or queries a sales database. Nothing here applies.

Except the mechanism, OpenAI’s discovery aside, was mundane failure. The preconditions were an agent with a goal, tools, and one mistake in the environment around it, and every production agent has all three. None of this is new doctrine either. The news is that four of the organisations best placed to know better failed the exercise anyway.

Notice what else the four incidents share, because it is the detail everyone is stepping past. Each one was a single mistake deep. One misconfigured service, one open route, one setup error, and the agent was on the real internet with nothing else in its way.

And something has changed underneath all of this. We are now building with models that find flaws quickly, a shift I wrote about in April when Anthropic’s Mythos model turned up a 27-year-old bug in one of the most heavily reviewed operating systems on earth. Finding holes used to cost an attacker time and expertise. It now costs API calls. Some unintended hole in your system is close to inevitable, and the software you are deploying is exactly the kind that finds holes.

So the design question is no longer how to prevent the breach. It is what happens in the minute after it. Two things need to be true when it comes. There must be more than one bulkhead between the agent and anything you cannot afford to lose, so that the breach costs nothing. And the breach must make a noise, so that you know it happened at all. Last week’s post was about the noise, choosing failure directions and signals you can actually hear. This one is about the bulkheads.

So count them, for each thing you cannot afford to lose, and keep counting until you know the number. The answer must never be one.

The tools are the boundary

The labs at least had a box to lose. Look at the actual shape of a production agent. The model runs in someone else’s cloud, and the tools run in your environment. There is no box around your agent, and there never was one to leak. Every tool you hand it is a hole you punched in your own boundary on purpose, because a tool that reaches nothing does nothing.

So the boundary is not a perimeter. It is a list, and the list is exactly what your tools can do.

That is where the long, hard thinking belongs, and it is harder than it looks. An agent uses everything you give it to complete the task you set, and it will combine those tools in orders you never designed and never tested. It is good at finding combinations that work. That is the capability you are paying for, and it does not switch itself off when the combination is one you would not have chosen. Nobody designed the route OpenAI’s models took through a package service and a research network into somebody else’s database. Every step was available, and the model was motivated.

So the question to ask of each tool is not what it is for. It is what that tool can reach on the day something upstream of it is wrong.

I can show you what that looks like, because our first mistake has already arrived.

When our first mistake arrived, it hit a bulkhead

We build an expert system for a client whose agent is heading, in deliberate stages, for autonomous work against production infrastructure. Two weeks ago that platform produced a real failure, and I wrote it up last week. A language model invented technical claims during content preparation, and every quality gate waved them through. A user was served confident, wrong words. We traced how they got in and fixed the pipeline that let them through.

That was the first mistake, live, in production, undetected for a while. Here is what it could touch. Nothing. Not because the model behaved itself, but because two bulkheads stood behind the wall that fell, and neither of them depends on the model behaving.

The first is staged autonomy. The agent does not yet hold the power to act on the client’s systems. We grant that power in deliberate stages, each stage earned once the one before it has proven out, and we are not at the acting stage yet. So a fabricated claim could mislead a reader, which is bad and was fixed. It could not run a command, change a setting, or touch a database.

The second is the registry. When the agent does earn the power to act, its boundary will be a literal list. Every action it can request comes from a registry, and anything not in the registry cannot be requested at all. A missing entry is a coverage gap, never a security hole. Seventeen operations are excluded on purpose, with written reasons. One backup operation was excluded because its file-path input would have been free text. Every character an input admits is a character an attacker gets to use.

And the data behind them is narrow. The agent holds no access to the client’s production database and no network route to it. That comes later, stage by stage, as the earlier stages prove out. What it can reach today is our own platform database, and there it can read what it needs and write to a small set of tables, logs and feedback among them. The blast radius of a fully manipulated agent is bounded by that list, not by how convincing its output is.

So the fabrication cost us a correction and an uncomfortable write-up, not an outage and not a breach. The mistake that happened was never the only thing standing between the agent and harm. That is the method. Put a bulkhead behind every wall, make it a different kind of control so one error cannot take out both, and count them before you count on any of them.

Run it before your agent does

The exercise is not a product you can buy. Write down the things a mistake must never reach. Then write down what your tools can reach once the first control fails, the credentials on the machine, the services that trust it, the routes out to package registries, code repositories and the open internet. Where those two lists touch with one control between them, that is the work. We have written before about fencing what an agent’s tools can do, and this is the other half, because a bulkhead you never hear fail is a bulkhead you are trusting blind.

Writing this post sent us back through all four of our own agent code bases asking exactly that question. It produced a work list, and most of it sat at the outward edge rather than deep inside, where the containment was already strong. That work has since been done.

The same sweep turned up something smaller. The phrase assume breach appears nowhere in any of the four. Not in a design doc, not in a commit, not in a policy. Yet the posture is everywhere, in the staged autonomy, in the registry, in the narrow list of tables an agent may write to. The discipline shows up in the commits, not the slogans.

You call it a sandbox. Your agent sees a list of tools and a task. Assume it leaks.