Stop asking the user to confirm everything

Anyone who has worked with a model knows the dial. Turn it one way and the model guesses. It fills the gaps, keeps moving, and you find out later what it decided on your behalf. Turn it the other way and it checks. Every gap becomes a question, every question needs an answer, and somewhere around the fortieth one you stop reading them.

Most teams treat this as a setting to tune. Ask a little less here, confirm a little more there. The advice that has hardened around spec-driven development over the last year says never assume, always ask. This month we ran a trial that followed that advice properly, and it broke in both directions at once.

The answer isn’t a position on the dial. It’s a different question.

Both ends of the dial, in one trial

Outline is a product we are building and trialling internally. You talk to it about something you want built, it interviews you, and it turns the conversation into a specification a coding agent can build from. In its first real sittings this month I used it to specify a file upload workflow.

The first version did what the consensus says. Nothing went into the spec until I had confirmed it. After two sittings it had produced 124 items waiting for my confirmation. Part way through the first batch I told it I was losing the will to live. A process that generates 124 things to confirm is not a process anyone sits through, and a process nobody sits through produces no spec.

Worse, the confirmations it did get were hollow. Reading the record afterwards, I found a point where I had said “Yeah. That matches.” The system recorded that as confirmation. The draft I was supposedly agreeing to had not been written yet. I was agreeing to the shape of a conversation, and the system took it as agreement to a document. Anyone asked to confirm 124 things says yes to most of them, and the yes stops meaning anything.

So that is the asking end. The guessing end showed up in the same trial. I had set a 5 GB upload limit. In one of the places the system was allowed to fill a gap on its own, it wrote an example scenario accepting a 10 GB file. A later plan, reading that scenario, asked politely whether 5 GB was really the limit. A rule I had stated plainly was being questioned on the strength of a number the system had invented and presented as if it were mine.

One trial, one dial, and it failed at both ends. Ask about everything and the person stops answering. Guess about anything and the guess becomes a fact the next stage builds on.

Ask by cost, not by category

What we changed was not the amount of asking. We replaced the question. Instead of deciding how much to check, the system decides each item on one test. Would getting this wrong materially change the outcome, or be costly to undo? If yes, stop and ask. If no, choose sensibly, record the choice, and carry on.

That test is the whole principle, and it is deliberately not a list. Money, access, data loss and the central steps of a workflow usually clear the bar. Spacing, navigation and loading behaviour usually don’t. But the bar is about consequence, not category, and context moves it. Sort order is a trivial choice on most screens and a serious one in a queue of urgent work. The system is expected to judge that in context, the way a good colleague would, and the rule carries one more sentence that does most of the work. Do not disguise a consequential assumption as a routine default.

The 10 GB file sits on the consequential side of that test. Not because it is a number, but because of what being wrong about it would cost. Under the old rule it was one gap among a hundred and the system was free to fill it. Under the new one it is a decision, and the system has to bring it to me.

The effect on the trial was immediate. The next plan assumed low-risk intent where it followed from what I had said, said that it had, and was usable. The handful of things I actually needed to decide came to me as decisions, with enough context to decide them, and I decided them.

A guess is never a decision

The cost test is a judgement call, and we let the model make it. What we don’t do is trust that judgement at the point where it matters most, which is when a guess is about to become something that gets built.

So the second principle is that the system may choose, but its choice is always labelled as its choice. It is never relabelled as something the owner decided. Example values come from the owner’s words or they are visibly illustrative, so that an invented number can never again pass as mine. And nothing consequential can be built on a choice the owner has not made. If the system guessed on something that clears the cost test, that guess cannot be accepted and the part of the spec it sits in cannot be marked ready. It comes back to the owner as a decision, with an example or a mockup to decide against, before anything is handed to a builder.

That is the answer to the obvious objection, that this is just deferring the questions. It isn’t. The questions that matter are still asked, but later, when there is something concrete to look at, and far fewer of them. The questions that don’t matter are never asked. The system makes a sensible choice and records it, so that if the choice turns out to be wrong it can be found and changed rather than discovered. Nothing arrives at the builder as a question. It arrives as a decision or a labelled default.

That is enforced by the system, not requested of the model. A rule in a prompt is a request, and models honour requests most of the time. Most of the time is fine for choosing a loading spinner. It is not fine for choosing an upload limit.

Two faults, two owners

We tested the spec against a coding agent three times. Once exactly as written. Once with one rule secretly deleted, every trace of it removed. Once with a fake rule secretly added and dressed up to look as if I had approved it.

The fake rule was caught. The agent refused to build it, because nothing else in the spec supported it. No example showed it in use, no screen or step depended on it, and its paper trail was a copy of a neighbour’s. The spec was complete enough that an invention stood out. That is what the principles above are for, and they worked.

The deleted rule was not caught. The agent read the rules either side of the gap, worked out what must have been there, and built it. It didn’t flag that as a guess, because it didn’t think it was guessing. It thought it was reading. I reviewed the result and didn’t notice either. We only knew because we knew what we had deleted.

That is the honest part of this post. A system can flag what it knows it is inventing. It cannot flag what it doesn’t know it is inventing. No amount of “say what you assumed” fixes that, because the agent genuinely believed it had assumed nothing.

So the two faults have different owners. A fake rule is the builder’s problem, and a builder that refuses anything the spec doesn’t support is the right defence. A missing rule is not the builder’s problem and the builder cannot be the defence. If a rule is missing from the spec, the interview failed, not the build.

Which gives the last principle. The only protection against a confident reconstruction of something nobody said is a complete record of what was said. The interview has to be driven by what a workflow needs, not by what the owner happens to remember on the day.

The system refuses what it can see

Stop tuning the dial. A system that asks about everything trains the person to stop reading. A system that guesses about everything trains the person to stop trusting. Neither is a setting, and the fix is not somewhere in between.

Decide each item by what being wrong would cost. Let the system choose where the cost is low, and make it record the choice. Where the cost is high, make it stop, and build the refusal into the system so that no model’s judgement, on a good day or a bad one, can quietly turn a guess into a decision.

Then understand what that buys you. The system refuses what it can see. What it cannot see, the rule nobody said, the thing everyone in the room assumed and nobody wrote down, is the interview’s job. Build the interview as if that is the only thing it is for.