When the human watching the AI checks out

When the human watching the AI checks out

An operator validates an AI's outputs. At first, they do good work. Then decent work. And very quickly, sloppy work. This isn't an AI critic talking. It's Amazon.

Eric Brandwine, vice president and chief security engineer at Amazon, put it bluntly to The Register in June 2026: "Humans are not very consistent. The human in the loop is not necessarily the gold standard."

The line sounds harmless. It actually takes aim at the central pillar of every reassuring pitch built around AI.

The safety net everyone is selling

"Don't worry, there's a human who validates it." You hear this the moment a company deploys an automated system on anything sensitive. A medical diagnosis, a credit decision, content moderation, an agent acting on live infrastructure. The human-in-the-loop has become the reflex argument for waving automation through.

It's also the magic phrase that unlocks budgets and approvals. A human keeps control, so the risk is contained. Except that human, watching correct output after correct output scroll by, eventually stops really watching.

Brandwine leans on a concept he has developed for years: the normalization of deviance. When shortcuts cause nothing bad for long enough, they become the new norm. The supervisor who waves through a hundred correct decisions will wave through the hundred-and-first the same way, including when it's wrong.

A mechanism known for forty years

The striking part is that research documented this long before ChatGPT. Human-factors specialists even gave it a name: automation complacency.

Raja Parasuraman and Dietrich Manzey wrote the reference synthesis in 2010, in the journal Human Factors. The finding fits in one sentence: the more reliable a system seems, the less its operator monitors it. Apparent reliability sabotages its own supervision. A safety net that works nine hundred and ninety-nine times out of a thousand is exactly the one nobody watches on the thousandth.

Mica Endsley described the aggravating side back in 1995: the out-of-the-loop problem. An operator who only validates gradually loses situational awareness. The day the system goes off the rails, that operator is the worst placed to take over, because they checked out long ago. We ask them to be a safety net, then install them in the exact position where they can no longer be one.

Aviation learned this the hard way. Generations of pilots ended up trusting the autopilot so completely that they stopped verifying it. MITRE's report on automation-induced complacency reduces to a chilling title: "Nothing can go wrong."

The self-driving car replays the same tune

For a more recent demonstration, assisted driving does the job. Real-world studies of drivers equipped with driver-assist features observe what researchers call passenger behavior. The car drives so well that the driver looks away, sometimes falls asleep. The instruction stays the same: "stay alert, ready to take back control."

The US road-safety authority itself noted that a mode disabling driver monitoring could "lead to greater inattention." We ask a human to continuously monitor a machine that almost always succeeds. Biologically, that instruction is untenable. Attention is not a tap you leave running.

Regulatory theater

And this is where it gets awkward. Europe made human oversight a legal obligation. Article 14 of the AI Act requires high-risk systems to remain "effectively overseen by natural persons."

The text is lucid, almost too lucid. It explicitly asks the supervisor to stay aware of their "tendency to automatically or excessively rely" on the machine's output. The regulator names, in black and white, the bias eating away at the safety net, then mandates the safety net anyway. We tick the "human oversight" box knowing the box doesn't stand on its own.

For the AI vendor, that box is a gift. It turns a structural flaw into a compliance argument. "A human validates" becomes a line in an audit file, not a protection measured in the field. The safety net exists on paper, which is often enough to reassure both the customer and the auditor.

What Amazon proposes, and what it avoids saying

Amazon doesn't stop at flagging the problem. The company proposes a shift: stop pretending a human validates every action, and instead trace accountability end to end. Each agent gets its own identity, every action is logged in the name of a human who answers for it, permissions are sliced finely. We no longer ask someone to watch permanently, we make it identifiable who has to own the call.

It's more honest. It doesn't solve everything. Tracing accountability after the fact is not preventing the error before it leaves. And the solution also suits a player selling AI at scale: fewer humans in the loop means less friction and more deployable automation.

It echoes a cousin of the problem we already saw at the individual level, when critical thinking erodes from outsourcing your judgment to a chatbot. Here it isn't a solitary mind sagging, it's the collective net meant to protect everyone. The same spring: a vigilance presumed permanent that never is.

The underlying question stays open, and it's not comfortable. If human attention checks out by design, then "adding a human in the loop" is sometimes set dressing rather than protection. And set dressing protects no one. It only reassures those watching from a distance.