Somebody has already run this experiment, with lives on it
Most organizations are approaching AI oversight as a new problem. It is not. Two industries met it decades ago under conditions where getting it wrong was fatal, spent thirty years on it, and reached a settled position before any of us had a model to supervise.
Aviation found that reliable autopilots were monitored less closely — not by the careless, but across the population, consistently enough that safety researchers named it and went looking for the mechanism. Medicine found that as diagnostic systems improved, clinicians accepted recommendations they would have challenged from a colleague. Different rooms, one shape, both documented in the 1990s.
Two failures, treated as one
The research that came out of those fields separates a pair of failures we tend to run together. The first is drifting out of watching something that is running — the autopilot is flying and you are no longer really monitoring it. Researchers call that complacency. The second is taking what the system hands you without checking it independently. That is automation bias.
From the outside they look identical: someone accepted an output they should not have. They are not the same failure, and they do not have the same fix.
Complacency is about attention over time. A system that is right almost every time slowly stops earning any. Automation bias is about authority in the moment — the output arrives already formatted, already confident, already looking finished, and it gets less pushback than a colleague would have got for saying the same thing less well.
Which is why a team can genuinely fix one and remain fully exposed to the other. Tighten the monitoring and you have done nothing about the moment the answer looks too polished to argue with. Teach people to challenge the output and you have done nothing about the process nobody is watching any more. Most of what gets called AI oversight addresses one of the two, and rarely says which.
A field with nothing to sell got here first
None of that research is about AI, and that is the whole reason to raise it. A field with thirty years, real consequences and nothing to sell arrived somewhere settled: relying on something that is usually right degrades your monitoring of it, predictably, in capable people. We are not at the beginning of understanding this. We are at the beginning of applying it somewhere new.
So the question worth putting to your own team is which of the two the last AI mistake actually was. And if it was the second, notice who was checking — automation bias is hardest to see in the person who wrote the prompt, because they already believe the shape of the answer.
Eighteen questions that show which of the two your team is actually exposed to: See where your team's oversight actually sits →
Keep reading: Ask when they last found something · Trusted AI
This is the thinking behind a Trusted AI Culture.
Sources: Parasuraman, R. & Riley, V., "Humans and Automation: Use, Misuse, Disuse, Abuse," Human Factors 39(2), 1997. Parasuraman, R. & Manzey, D., "Complacency and Bias in Human Use of Automation: An Attentional Integration," Human Factors 52(3), 2010.
Written with AI in the loop: my idea, AI drafted and sharpened, my judgment on the way out. Every word is mine to stand behind. — Darren