Stop choosing models by leaderboard

Six weeks after shipping Claude Fable 5 at $10 per million input tokens, Anthropic released Claude Opus 5 at exactly half the price. The cheaper model arrived winning. It leads Fable 5 on 7 of the 12 benchmarks the two models share, including a 9.6 point gap on Frontier-Bench, and it does it at $5 in and $25 out.

If you choose models by leaderboard, the decision is over. The newer, cheaper model wins, switch everything, bank the savings.

Except the early practitioner reaction, mine included, is that Fable still feels like the smarter model on hard, ambiguous work. And when we ran both models against an evaluation built from our own production workload, the leaderboard result inverted.

The gap between those two pictures is not a mystery. It is a measurement problem, and it has a practical answer.

The scoreboard measures training, not intelligence

Almost every public benchmark is a verifiable-reward task. Exact match, passing tests, an answer a script can check. That is not a coincidence, it is what makes a benchmark a benchmark.

It is also precisely the surface that reinforcement learning post-training optimises. RL needs a grader, benchmarks are made of graders, so a benchmark score is a better proxy for how much recent post-training a model has absorbed than for how capable the underlying model is.

Here is why that favours the cheaper model. RL cycles on very large models cost more in dollars and wall-clock time per iteration. The constraint is economic, not learning-theoretic, larger models are typically more sample-efficient per episode. But economics decides cadence, so the smaller flagship absorbs more environments, more graders, and more RL compute per quarter. Opus 5 shipping six weeks after Fable 5 with newer post-training and higher launch scores is exactly what that dynamic predicts.

One launch is one data point, so I will state this as a prediction the post can be held to. Over this model generation, watch the point releases. The smaller flagship should pull further ahead on benchmark scores while the bigger model keeps its edge on work no script can grade.

The strongest evidence already sits in the launch tables. Look at where Fable’s remaining leads are. SWE-bench Pro, the hardest coding suite. The longest-horizon autonomous agent tasks. Where a benchmark is difficult enough to resist saturation by post-training, base quality shows through even on the scoreboard.

One reading-comprehension note for anyone buying off these tables. Several headline figures published alongside Fable 5 are actually Mythos 5 figures, the configuration without Fable’s additional safeguards, not measurements of the public model. The comparison is muddier than the leaderboards present, which is one more reason not to let them make your decision.

The vibes are contaminated too

Before I lean on “Fable feels smarter,” I should concede how weak that evidence is.

People hand the expensive model their hardest, vaguest problems, exactly where base quality shows, and they expect it to be better. Familiarity makes it worse. Practitioners have had seven weeks with Fable and under one with Opus. “Everyone who used both prefers Fable” cannot be a settled consensus six days after launch. It is first impressions measured against a familiar baseline.

So neither the public benchmarks nor the community vibes will tell you which model to run. You need evidence of a third kind.

Ask what kind of grader the task has

The decision rule I would give any technical leader starts with a question. Does this task have a grader a training loop could use?

If the answer is a checkable one, produced in high volume, verifiable in milliseconds, the heavily post-trained model wins, and its per-step reliability compounds across long agentic loops. This is why “just buy the expensive one” is actively wrong for high-volume agent workloads, not merely wasteful. In our RAG platform, the relevance judge that makes thousands of binary calls runs on Haiku for exactly this reason, while the conversational agent, where the failure mode is technically correct but misses the point, gets the frontier model.

Notice the phrasing, though. A grader a training loop could use. Not a task you can mark. The distinction matters, and our own evaluation is the proof.

Our knowledge base platform has a feedback mechanism. When a user flags an answer, a triage agent investigates. It has the conversation, the retrieval pipeline, and every tool call to hand, and it works out whether the problem is content missing from the corpus or a fault in retrieval, then suggests a fix. We keep every historic case, and for 45 of them we know what we actually did to fix the problem. That gives us a private benchmark. When a new model ships, we rerun the triage process locally against those 45 cases. Both models diagnose the issue almost perfectly, that part of the task is saturated. The score is on the proposed remedy, judged against the fix we actually implemented, and that call is not black and white. There is not always one right fix, so a human judges whether the model’s proposal would have worked at least as well as ours did. That is a feature, not a weakness, the eval is judgement-shaped because the task is.

Fable 5 proposed an acceptable remedy in 43 of 45 cases. Opus 5 managed 37.

The model that trails by 9.6 points on Frontier-Bench beat the benchmark winner by six cases on our workload. And here is the instructive part. Triage is gradeable, we can say with confidence whether a proposed fix would have worked. But it is not verifiable-reward shaped. Reaching a good remedy means weighing ambiguous evidence across a conversation, a retrieval pipeline, and a chain of tool calls, then deciding what to do about it. It is judgement with a scoreboard attached, and that is where base capability shows.

A week with both

My own use over the past week matches the eval more than the leaderboard, with honest caveats.

For coding I cannot yet tell them apart, though I am not sure I have thrown a genuinely hard problem at both. In practice I find myself using Opus as the daily driver and reaching for Fable when I want something analysed deeply. That habit, formed without thinking about it, is the decision rule operating at the level of one person’s chat window.

Writing is where it gets interesting, because I almost prefer Opus. Fable produces less accessible prose, it assumes reader knowledge, and I sometimes struggle to follow its train of thought, unpicking its arguments to bring them round to my own mental model. Maybe that is a measure of my own shortcomings rather than the model’s. Either way it is a useful reminder that the smarter model is not automatically the more useful one, even on unscoreable work.

Build your own leaderboard

The public benchmarks will not settle this, and the vibes cannot. The only evaluation that answers the question you actually have, which model should run my workload, is one built from your workload.

Ours is 45 cases, assembled from feedback we were handling anyway, with the fixes we actually shipped as the ground truth. Rerunning it takes an afternoon, and while the scoring involves judgement, it is our judgement about our workload, which is precisely what a launch table can never give you. It is the reason we know, rather than believe, that our triage agent stays on the bigger model while our high-volume judges do not.

You do not need a research team to do this. You need a handful of real cases from your own system, verified answers, and the discipline to rerun them every time a launch table tempts you to switch.

Opus 5 at half the price is a genuinely good model, and for most gradeable, high-volume work it is probably the right one. But if you are choosing by leaderboard, you are measuring a vendor’s training investment, not your own outcomes. Build the eval. Let your workload cast the deciding vote.

If you are wrestling with model selection across a production agentic system, talk to us. We have made most of the mistakes already.