Choose the direction your AI fails, before it chooses for you

Your agent will not crash. It will tell you a story.

When the UK AI Security Institute’s evaluation agents took 19 unauthorised actions across 122 test runs in July, nothing in the agents’ own behaviour surfaced it. A network egress alarm did. When Anthropic disclosed that its models had breached three real companies during testing, the telling detail was buried in the write-up. The victims did not know. Anthropic found the incidents by reviewing 141,006 evaluation runs.

Everyone is reading these as stories about what agents can do. Read them again as stories about detection. A government institute and a frontier lab caught agent failures with a perimeter alarm and a manual review. The failing systems’ own signals said nothing.

The failure that stays green

Forty years of operational practice assumes failure is observable. Software throws, crashes, trips a health check, and monitoring, alerting and on-call are built on that. LLM components break the assumption. Their characteristic failure is not an error, it is a fluent, plausible, wrong answer, delivered with the confidence of a correct one.

A June arXiv paper, When Errors Become Narratives, names the class, fail-plausible. It is one practitioner’s postmortem of a single production system, not peer-reviewed research. But across eight weeks, roughly 70 percent of its silent failures were caught by a human looking at output. Its 4,286 unit tests and 827 governance checks caught approximately none, and stayed green throughout. Silences ran from 13 hours to 60 days. One system only, but the same fortnight’s news, and our own incident log, show the same pattern.

Enterprises are answering with detection budgets, LLM observability is a $2.69 billion market growing at 36 percent. Yet in Gravitee’s survey of 919 executives, 88 percent reported an agent security incident in the last twelve months, only 21 percent have runtime visibility into their agents, and 82 percent believe their policies protect them.

That money is pointed at the wrong end of the problem. I can say so directly, because last week a client showed us the wrong end.

Fabrication walked through every safeguard we had

On 1 August an email arrived from a principal at a client whose expert knowledge platform we build and run. A coding-agent session had served him two factual errors, both verbatim from the corpus. The answering model was faithful to retrieval, retrieval was faithful to the corpus, and the corpus was wrong.

The errors entered upstream, at a preprocessing step whose entire purpose was hygiene. Source transcripts and internal documents were reshaped by a language model before loading, turned into narrative, stripped of customer references. That responsible-handling pass is where the fabrication got in. One chunk invented an expansion for a benchmark acronym and recommended a tuning value far beyond what the vendor’s own guide, elsewhere in the same corpus, calls unusual, dressed in precise numbers that traced to nothing. Another silently dropped the scope of a true claim, turning it into a false general one, which the principal falsified by live experiment on two product versions.

Our chunk quality gate scored the fabricated chunk 4/5/5 and praised its concrete performance data. Genuine, appropriately hedged expert prose scores in the same band. The gate cannot tell fabrication from expertise, because form is all it can see, and there is nothing to tune. Groundedness checks passed by construction, since an answer that faithfully quotes a wrong corpus is a faithful answer. At the time, 44 percent of the live corpus was LLM-synthesised prose no expert had verified.

Diagnosis took under an hour, tracing the conversation, identifying the chunks, establishing provenance. A remediation plan took two or three hours more, an engineer built it within days, and it is in final review as I write. The speed is not the point. The point is what the fix is not.

The fix is a direction, not a detector

We did not buy a better detector. Ours had worked exactly as designed, that is the problem. Detection grades what it can observe, and a fluent wrong answer presents all of it as normal. Detection is necessary, we run plenty of it, and it is the second line. The first line is deciding what happens when it misses, because it will.

So the fix is a failure direction for the corpus itself. Synthesised chunks whose claims can be neither verified nor refuted are quarantined, excluded from retrieval until a human approves them, and nobody is required to staff that queue. The adversarial judge cannot mark a claim verified, because general-knowledge agreement is not verification, so nothing becomes retrievable on a model’s unsupported say-so. Provenance detection can escalate a document into the trusted tier, never downgrade one out. And the quality gate’s old bias is inverted, specific numbers without traceable provenance now lower confidence instead of raising it.

None of this was invented under pressure. Every LLM component in our retrieval pipeline already had a documented failure direction, marked intentional, do not normalise. The retrieval gate fails towards searching, extra work is safe, silently skipping retrieval is not. The relevance judge fails towards discarding everything, injecting nothing is safe, injecting something misleading is not. Even self-reported confidence is a two-value enum, high or medium, with medium always routing to the more conservative path, because finer buckets are an illusion of signal.

We have no documented instance of those directions firing in production. They may have fired without anyone noticing, and that ambiguity is the point. A cheap-safe failure does not demand attention. That is its job.

When the safe direction is wrong, you hear about it

Cheap-safe is not free, and we have paid for it. We hold documented feedback that reads, in effect, I got nothing valuable from this and I expected to. We went into the corpus and found the content was there, discarded by the relevance judge. That feedback loop is where much of this architecture came from.

Notice the shape of that failure. When discard is wrong, users notice the absence and tell you, and every complaint is actionable. When fabrication is wrong, users get a confident, fluent answer and have no reason to complain at all. The failure direction decides whether your feedback loop functions. A system that fails towards absence generates complaints. A system that fails towards fabrication generates satisfaction.

We have seen the same shape in testing, a threshold set on the wrong scale silently filtered every retrieved chunk and the system answered from nothing, and a human caught it before it shipped, because absence is noticeable in a way fluency never is.

Once, the direction saved us outright. Our interview platform’s flush cleared a message buffer before confirming the database write, so failed writes silently lost an expert’s words. No test caught it. The interviewee did, reviewing the generated narrative as a designed step in the workflow, the paper’s 70 percent statistic inside our own walls. Nothing was lost, the original recording survived downstream and we reprocessed it. The failure fell towards recoverable. That is the entire discipline in one bug.

Write the direction down

For every LLM component you run, decide which way it falls when it is wrong, towards cheap-safe or towards silent-dangerous, and enforce the choice in structure rather than instruction. A prompt that says be careful is instruction. A judge that cannot emit the word verified is structure.

The costs are real, and they should be priced rather than hidden. Searching unnecessarily costs latency. Discarding costs recall. Quarantine costs coverage, and our spec owns in writing that genuinely novel knowledge will often arrive unverifiable and wait for a human. Those costs are chosen and visible. The cost of silent-dangerous is neither.

If you cannot name the direction one of your components fails, you have already chosen for it. It fails silently, and when it does, it will not crash. It will tell you a story.