The Substituted Question: Why Your Review Says Yes Every Time

You closed the laptop at six on a Friday with eleven pages of notes, three decisions and one realisation you had not had that morning. It was a good session. You did not need to be told that it was a good session; you could feel it.

Then, months later, you opened last year’s notes.

The same constraint. Different vocabulary. Four quarterly reviews between the two documents, every one of them honest, every one productive, and the thing you had been reviewing had not moved.

The standard reading of that experience is that you were not being honest enough with yourself. I want to argue something less comfortable and more useful: you were entirely honest, the review worked exactly as built, and it answered a question you never asked.

A scale can only weigh what you put on it

Consider what was actually in front of you at the moment you judged that session.

Notes. A revised priority list. Three decisions. One uncomfortable realisation. All of it real. None of it invented. Every item genuinely produced by three hours of difficult thinking that most people avoid.

Now consider what was not in front of you: your actual position.

It could not have been. Your position is not the kind of thing that appears on a table at the end of a session. It is an absence right up to the moment it changes, and by the time it changes the review is months behind you. So at the instant you assessed whether the review had worked, the evidence for yes was vivid, countable and sitting on the table, and the evidence for no was nothing whatsoever. There was no pile of not-progress on the table to weigh against the notes.

There was no pile of not-progress on the table to weigh against the notes.

This asymmetry is not a defect in your character and it is not specific to founders. It is one of the more thoroughly established findings about how attention operates. In 1980, Newman, Wolff and Hearst ran six experiments across varied materials, procedures, kinds of feedback and instructions, and established in adults what had already been demonstrated in animals and in young children: we learn to detect the presence of something far more readily than its absence. What is there registers. What is missing barely does.

That is a laboratory finding about discrimination learning, not a study of business reviews. What it gives us is the mechanism’s basis, not its measurement. But the basis is enough to predict what happens next in a room where one side of the ledger is visible and the other is structurally invisible.

A review conducted in good faith, by a person with nothing to gain from flattering you, returns yes.

The swap

Daniel Kahneman and Shane Frederick named the move in 2002, in their chapter on attribute substitution. When the property you are trying to judge is difficult to assess, the mind reaches for a related property that is easier to assess, and answers about that one instead. The word in their description that carries the weight is this: people make the substitution unwittingly. There is no sensation of having switched. That is not a footnote to the phenomenon; it is the phenomenon.

So there are two questions in the room, and only one of them gets answered.

Sitting in the chair, those feel like the same question. They are not even close. One is about a level. The other is about an event. You can generate events indefinitely without ever changing a level, and the events will keep producing the evidence that the level is fine.

You did not fail the review. You passed a different review, and never saw the swap.

You can generate events indefinitely without ever changing a level, and the events will keep producing the evidence that the level is fine.

What makes this more than a metaphor

Two pieces of evidence turn this from a tidy framing into something you can act on, and they do different jobs.

The first establishes that in humans, on real goals, the felt read and the measured read genuinely come apart.

Smyth, Milyavskaya and colleagues pooled four datasets covering 351 people pursuing academic goals and weight-loss goals, and compared what participants said about their own progress against what could be objectively measured. Across the studies, the subjective and objective measures shared between 5 and 39 per cent of their variance. Related, then, but nowhere near the same thing. Their own conclusion is put more bluntly than I would have dared: subjective and objective measures “may not assess the same thing and should probably not be used interchangeably or interpreted as proxies for one another.”

Academic and weight-loss goals are a narrow setting, and the narrowness is exactly what makes it transfer. Those are goals with a bathroom scale attached. An external instrument is physically present in the room, and the person can step on it whenever they like. If the felt read and the measured read drift that far apart under those conditions, they do not drift less on a goal that has no scale in the building.

The second piece of evidence does a different and more surgical job. It kills the fix that everybody reaches for.

The better reviewer does not help

Once the problem is visible, the instinct is immediate and universal: improve the reviewer.

Be harder on yourself. Bring in an adviser who owes you nothing and is not paid to be pleasant. Tell the AI to challenge you instead of agreeing with you. Set a higher bar. Book the review with someone who will push.

Every one of those moves raises the standard on the substituted question. Not one of them restores the original.

That claim rests on the mechanism above: presence registers and absence does not, so no amount of improved judgement puts your position back on the table. What follows is not proof of that for humans. It is a controlled illustration of the same structural principle, in a domain where it could be measured directly.

Look closely at what a harsher reviewer is doing: asking did this session produce something good enough rather than did this session produce something. That is a higher threshold on the same event. The level is still not on the scale. Nobody in the room has gained access to a reading of your position, because the reading does not exist in the room to be accessed.

There is now a controlled test of precisely this proposition, and it arrives from an unexpected direction. What it shows is narrow, and over-claiming it would be fatal to the argument.

Park and Choi ran a preregistered measurement study on long-running autonomous software agents. The design is the point. They held the agent and its tool surface fixed, and varied one thing only: what the evaluator gating the loop was grounded in. Across 54 cycles, the agent claimed improvement every single time. Fifty-six per cent of those cycles had a measured change of zero or worse. Left to grade itself, the gate collapsed into accepting everything, and eroded the best state the system had reached, by 19 per cent.

Then they tested the obvious fix, properly. They gave the evaluator everything it could want: the full artefact text, the complete record of what had changed, and its own history of every verdict it had previously issued. A strong, well-informed, structurally separate judge.

It accepted cycles of which 44 per cent were real-world regressions. It also rejected 38 per cent of the genuine improvements. Their registered hypothesis had been that a strong judge closes the gap. It was rejected.

A harsher reviewer is not a conservative error. It is not an error in a direction at all. It is noise.

Read those two numbers together, because that is where the sting lives. The fallback position on “be harder on yourself” is that it is the safe failure: worst case you are too strict, and being too strict never ruined anybody. The data does not support even that. The judge was wrong in both directions at close to the same rate. It threw out real progress almost as readily as it waved through regression.

And then the finding that matters most. On a task where success could be verified from the artefact itself, the false reporting vanished entirely, and the gap closed to within the threshold they had registered in advance.

Nothing about the judge changed between those two conditions. Its independence, its access, its rigour, its information: all held constant by the design. The only thing that moved was where the success signal lived.

I am not claiming you are a software agent, and that study cannot tell you anything about human psychology. It was never asked to. It does exactly one job here and it does it cleanly: it is a controlled test of whether the quality of the evaluator is the variable that determines the gap. It is not. The human mechanism stands on its own evidence, above.

Why adding more reviewers does not rescue it

When a harder critic stalls, the next move is redundancy.

An adviser. A quarterly scorecard. A peer group. An AI thinking partner. Perhaps a board. Four or five independent readings must be safer than one, and this is such an obvious improvement that it rarely gets examined.

They are not independent in the way that matters. Ask what each of them actually reads.

Your adviser reads your account of the quarter. Your scorecard reads the numbers you selected for it. Your peer group reads the version of the situation you can describe out loud in twenty minutes. Your AI partner reads the framing embedded in the question you typed. Every one of those inputs originates inside the same room, and passes through the same account.

Engineers who build safety-critical systems call this common-cause failure. Redundant components that were assumed to be independent fail simultaneously because they share a vulnerability nobody knew was shared. The redundancy does not merely fail to protect; it conceals the exposure, because five green readings feel like far better evidence than one.

You have not built five instruments. You have built five copies of the same instrument, all pointed at the same place, all of them reading in-band.

Where this sits against the Validation Spiral

I have written before about a related mechanism, and this piece corrects part of it. Saying so plainly is better than hoping nobody notices.

The Validation Spiral describes a four-phase cycle in which capable people examine their lives with excellent instruments, generate genuine insight, and produce no structural change, because the identity thermostat filters every insight for safety before it reaches the decision. That mechanism is real and I stand behind it. Its first prescribed countermeasure is to state the target out loud before a session: optimise for my growth, not my satisfaction.

On the evidence above, that countermeasure is a better in-band judge. And better in-band judges do not close the gap.

The reconciliation is not a retraction, because the two mechanisms sit at different depths. The Validation Spiral is about insight that fails to convert into change. The Substituted Question is about something upstream of that: whether you can tell, at all, that conversion did or did not occur. You can have the spiral running without the substitution. You can have the substitution running without the spiral.

But if the substitution is running, you cannot detect the spiral. Detecting it requires precisely the external reference the substitution has removed. Which means the Validation Spiral’s countermeasures are not wrong; they need one thing installed beneath them before they can be evaluated at all. This piece names that thing.

What contradicted a verdict about me

In 2011 the paralysis took my legs within seven days, descending from the navel. Three days later it began climbing, up from the navel toward my chest, until I was breathing with only the top of my lungs. A ventilator was anticipated.

I have carried that forward into how I have run everything since, and the structure holds whether the subject is a body or a business. The review is not the problem. The review’s inputs are.

The sentence to write before the next review

Before the session. Not after. One line.

Name the observable that would have to change, stated so that someone outside the room could check it without asking you.

The last clause is the whole test, and it is stricter than it sounds. “Clarity on the direction” fails it. “Better understanding of the constraint” fails it. “The team is more aligned” fails it. Each requires you to be the instrument reporting on yourself, which returns the problem to its starting position.

What passes: a figure in an account. A name on a signed agreement. A date on a document. A decision that now sits with somebody else and can be observed sitting there. A number of hours in a calendar that a person who has never met you could count.

Two outcomes, and both are worth having.

You can name it. Then the success signal is out of the room, which is the entirety of the intervention. Write it before you sit down. Written afterwards, it describes what happened, and a description is not a test. Written beforehand, it is a claim that can turn out false, which is what makes it worth anything.

You cannot name it. That is the finding, and it is a real one. It does not mean the session was worthless, that the insight was fake, or that you were lazy. It means the review had no access to the quantity it was supposed to be reporting on, and that its yes was never a statement about your position at all. That is worth knowing about a process you have been running for years.

What is already working

None of this argues for fewer reviews, and stopping would be a poor reading of it.

Building a recurring review at all is the expensive half, and you already own it. It is easier to run on instinct and a quarterly alarm. It is easier still to schedule the thing and then stop attending it in any real sense while the invitation stays in the calendar. You built it, you have kept it, and you turn up to it honestly enough that finding last year’s notes bothered you. The insight it produces is genuine. The discomfort is genuine.

What is missing is one sentence, written before you sit down.

The instrument is built and it is expensive and it is yours. It is pointed at the wrong quantity. Turning it costs less than anything else you have already done.