Confidence is not a Control

Just because it looks green and fuzzy, does not mean it is actual governance

Report this article

Martijn Veldkamp

Published Oct 19, 2026

+ Follow

Imagine an agent recommending something and adding “94% confident” underneath it. Nice green number, perhaps a little shield icon. You know, the sort of screen that makes everyone feel a bit more comfortable.

Who decided it means anything?

That number might come from a model trained to estimate a specific outcome. It might be a score calculated by another component. It might simply be text the language model generated when somebody asked how sure it was. Those are very different things wearing the same reassuring percentage sign and usually the colour green.

A percentage of what, exactly?

I wrote in The Benchmark was never about you a while ago about the distance between a general result and the work your organisation actually needs to do. Confidence introduces another opportunity to cross that distance without noticing.

A score can rank likely outcomes usefully without the number representing a reliable probability. Even a well calibrated probability needs a clear event attached to it.

“94% confident” for a payment is based on? That the invoice matches a purchase order? That the bank account belongs to the supplier? That the supplier still wants payment to that account? You can answer nay question correctly and still send the money to the wrong place.

If you use a score to authorise action, somebody needs to establish what it measures, how it behaves on representative cases and which mistakes remain. Otherwise the colour green is doing a surprising amount of governance.

Kermit would like a word about the workload. And yes, this is where my brain connected green confidence numbers and the Muppet show.

The consequence has an owner

Take a hypothetical assistant drafting product descriptions. A small error may be easy to catch and cheap to correct. Now give a similar assistant authority to change a customer’s entitlement or reject a claim. The same apparent accuracy can support a very different level of risk.

The Thirty-Five Minutes Nobody Counts: Why Your Impact…

Crises Control

1 month ago

Good Slow, Bad Slow

Vivek Bapat

1 month ago

💙 Edition 16: Nobody Notices Because the Output Looks…

Julia Chin 陈碧茹

1 month ago

My point is that probability and consequence meet inside a business decision. The model cannot settle the consequence by sounding more certain. Somebody has to decide which errors are tolerable, which actions can be reversed and what protection the affected person receives.

That decision belongs to an accountable owner with authority over the process. It needs input from the people who understand the system and the people who live with its mistakes. A developer should not have to invent the organisation’s risk appetite by choosing a threshold because the configuration requires a number.

I worry that this is precisely how some thresholds get chosen. The team raises the value until the demo stops making obvious mistakes, lowers it until enough work flows through and calls the result a control. It might be a reasonable experiment. It becomes a control only when the organisation has evidence for what it permits and a response for what goes wrong.

Give the number something useful to do

I am not arguing that confidence scores are useless. A validated score can help prioritise review, narrow an action’s scope or identify cases where the system should stop. It earns that role through evidence about a particular task.

For architects, the design question is where that evidence enters the action path. A recommendation can pass several checks and still face a hard limit on value, recipient or frequency. For example with payments: A changed bank account can require independent confirmation regardless of how confident the agent feels. A service can stop when inputs fall outside the conditions used to validate it.

Those boundaries need enforcement. Writing them into an instruction file is a starting point, and I have written plenty about the value of context. But a component with certain permissions should still face a boundary that still holds when its reasoning is wrong.

For developers, that means keeping the score separate from permission. For business owners, it means describing the consequence they accept and the remedies they fund. For a CTO, it means asking whether the dashboard reports a measured capability or offers reassurance that nobody has connected to an operating decision.

The distinction matters during an incident. You need to know which action was authorised, under which conditions, with which evidence and by whose policy. “The model was quite sure” will make a poor incident report, even with the shield icon.

My opinion is that the next useful AI governance discussion starts after the percentage. Assume the system gets this particular case wrong. Can we contain the effect, find the decision and put the customer right?

If you can answer that, the number may have a useful job. If you cannot, changing 74 to 94 will mainly improve the colour of the slide.

Confidence is not a Control