Is a zero exception rate always a red flag?

Not always, but it always warrants explanation. A mature process with strong upstream controls may genuinely produce few exceptions. The distinction is whether the organisation can explain the zero — or has simply never looked.

Does this apply where the model only recommends?

Yes, and often more so. Where a model recommends and a human formally decides, the accountability sits with the human while the reasoning sits with the model. That gap is exactly what oversight testing is for.

How do you test whether human oversight actually works?

Why the approval step is not the evidence

On a process map, human oversight almost always appears as an approval box. Walkthroughs confirm the box exists, screenshots go in the file, and the control is marked as operating.

None of that tests the control. An approval step evidences that someone clicked. It does not evidence that they could have declined.

This is a familiar problem in internal control work, and the AI context does not change its shape — only its scale. The same question applies: does the person exercising this control have the authority, the information and the time to exercise it?

Six things worth testing

1. Is the approval ever exercised against the model?

Pull the exception rate. A control that has never produced a rejection in twelve months is not a healthy process — it is usually a control that is not operating. Ask what happened in the cases that were rejected, and whether anything followed.

2. Is the approver independent of the outcome?

Check whether the approver owns the process, and whether declining carries a cost for them — throughput targets, turnaround times, or a manager whose numbers depend on volume. Where declining is expensive, the control degrades quietly within a quarter.

3. Does override authority exist, in writing?

Approval alone leaves two options: accept, or block the process. Test whether a defined override path exists, who may use it, and whether it has ever been used. An unused override right is a claim, not a control.

4. Is there a decision trail?

Look for who decided, on what basis, and when. If the record captures only the outcome and not the inputs the reviewer saw, the control cannot be re-performed — by you or by anyone after you.

5. Does time pressure hollow the control out?

Compare the time allocated to the review against the weight of the decision. Where a step is expected to clear in seconds, no amount of documented authority makes it a control.

6. In the sample, did the human actually challenge the output?

This is the test that settles it. Take a sample and look for evidence of engagement — a comment, a correction, a query raised, an input checked. Uniform approvals across a sample tell you the control is decorative.

What a failing control looks like

The pattern is consistent: the approval step is present, the documentation is complete, the exception rate is zero, the reviewer is not the process owner, and the record shows outcomes without inputs. Each element looks acceptable alone. Together they describe a control that cannot operate.

A practical first pass

List the processes where a model influences the decision directly. For each, score authority, competence, time and evidence. In most organisations this produces a clear gap table within the first week — and it is the input an AI governance programme actually needs before anyone writes a policy.