What happens when human oversight breaks down in practice?

Human-in-the-loop sounds reassuring. In practice, it breaks down for well-documented reasons.

Approval fatigue: after the 20th prompt, people start clicking “yes” without reading. Knowledge gap: most users cannot evaluate whether a command or action is safe. Speed versus safety: prompting for every action defeats the purpose of having an agent.

This isn’t speculation. Over fifty years of automation research — Don Norman and others — confirm that highly trained people cannot reliably monitor automated systems for long periods and then rapidly take control when something goes wrong. The better the automation, the less vigilant the human. It’s the paradox of oversight: the more reliable the agent, the less likely the human is to catch the failure.

If your safety model depends on vigilance, it will fail when vigilance fades. That’s not a risk to manage — it’s a design flaw. The alternative is structural containment: define what the agent can do, and let architecture enforce the boundary rather than relying on a human to watch every step.

Go deeper: AI Agents Need the Inverse of Human Trust argues that as reliability improves, risk may actually grow — because humans stop watching.