What does oversight even mean once an agent has already acted?

← Back to Blog

“The Oversight Fallacy: Why AI Agents Require More than Humans-in-the-Loop”

Passi & Singh (2026)

In a field study of developers supervising coding agents, the supervisors did not review everything. They built heuristics for what to check and what to let through, settling for efficient oversight rather than exhaustive oversight under real workload constraints (Dhanorkar et al., 2026). That reads as a discipline problem. It is closer to a capacity problem. People handed more output than they can inspect will ration their attention, and the rationing rule is usually invisible to whoever designed the approval prompt.

Consider a hypothetical. A procurement agent reconciles invoices overnight, touching four systems and posting several hundred entries before anyone logs in. The dashboard shows green. A reviewer scans a summary, approves, moves on. Nothing in that interaction tells her which of those entries deserved her judgment, and nothing gives her a way to undo entry 214 once the payment has cleared. She has performed oversight. She has not exercised it.

My argument is that oversight has to become a structural property of the workflow rather than a person stationed at the end of it. That means defined escalation thresholds, specifying when an action stops and waits for a human decision, and defined recovery rights, specifying how a wrong action gets contained or reversed after the fact.

The Research

Trust architecture is the deliberate design of transparency, feedback loops, and calibration mechanisms that let people rely on AI systems to the right degree. Agents break most existing trust architecture because the calibration signal arrives after the action rather than before it. Narayanan and Feigh (2026) found that dependency between AI teammates masks less severe errors from human supervisors: when one agent's output feeds another's, the interface surfaces the loud failures and buries the quiet ones. The supervisor's confidence stays high precisely where her information is worst.

The decision-science scaffolding predates the current agent wave. Parasuraman, Sheridan, and Wickens (2000) gave the field its taxonomy of automation levels and the intervention points that correspond to them, and graduated-oversight designs still build on that structure. What changed is throughput. Passi and Singh (2026) document how agents acting across many systems at once let a small mistake cascade through connected workflows faster than sequential human review can track.

The second-order effect worries me more than the first. Singh and Passi (2026) argue that the judgment oversight requires is learned through repeated practice, and delegation to agents removes the practice. Mitchell, Ghosh, and Passi (2026) make the parallel claim that the cognitive capacities oversight depends on degrade under extended use of agent systems. Together they describe a system whose reviewers become less qualified to review the longer it runs. That is trust architecture failing in slow motion, and no number of additional approval prompts will catch it.

What This Means in Practice

Start by replacing the single approval gate with a routing function. Kang (2026) proposes a three-tier model for agentic code generation in regulated domains, using a deterministic classifier to route actions to human-in-the-loop, human-over-the-loop, or automated-with-monitoring treatment. The move worth stealing is that the routing rule is written down and auditable. Someone decided, in advance and on the record, which actions stop for a person. A generic "approve?" prompt makes that decision implicitly, several hundred times a night, using the reviewer's fatigue as the algorithm.

Reversibility belongs in that rule as a first-class variable, a case I made earlier. Kang (2026) treats it as one, routing irreversible actions to stricter tiers because rollback is not otherwise available. That split runs through an agent's whole action space. Posting a draft or re-running a query is cheap to undo. Sending an external communication or releasing a payment is not. Zhu, Lu, Ming, Lee, and Wang (2025) catalogue mechanisms for the second category, including circuit breakers that halt a process mid-run and appeal bundles that package the evidence needed to contest an action after it lands.

The organizational-psychology piece is the one leaders skip. If reviewers ration attention under load (Dhanorkar et al., 2026), adding oversight duties without subtracting other work produces the appearance of review and none of the substance. Oversight capacity is a staffing decision with a number attached, not a value statement in a policy document.

Three Things to Take Away

Define escalation thresholds, not approval prompts

Specify which action classes stop and wait for a human decision, and write the rule where it can be audited and revised. Kang (2026) demonstrates a deterministic classifier doing this routing across three oversight tiers, and Zhu et al. (2025) catalogue the supporting signals, including confidence scores and policy attribution, that make an escalation legible to whoever receives it.

Build recovery rights into the workflow

Assume some wrong actions will execute; the design question is what happens next. Circuit breakers and appeal bundles let a workflow contain or reverse an action without depending on a human noticing in time (Zhu et al., 2025). Where reversal is impossible, that irreversibility should be what triggers the stricter oversight tier (Kang, 2026).

Treat oversight skill as perishable

Judgment for oversight is built by doing the underlying work, and delegating that work removes the practice (Singh & Passi, 2026). Extended use of agent systems appears to degrade the cognitive capacities oversight requires (Mitchell et al., 2026). Design checkpoints where reviewers do the task rather than rate it, and monitor rubber-stamping rates as a metric.

My Two Cents

An approval prompt with no realistic capacity behind it is a liability transfer dressed as a safety mechanism. It moves accountability onto the person who clicked without giving that person the time or the authority to have decided otherwise. A fair number of current agent deployments are doing this, and I doubt the people designing them intend it. The prompt feels like control because clicking feels like deciding.

The better question for any leader deploying agents is not "who approves this?" but "what happens at 3 a.m. when this goes wrong and nobody is watching?" If the honest answer is that it compounds until morning, the oversight design is decorative. Recovery rights are less satisfying than approval gates because they concede that mistakes will land. They are also the only part of the architecture that still works when the reviewer is asleep.

Read to Learn More

Narayanan, R., and Feigh, K. M. (2026). Designing for Oversight. An empirical study of how dependency between AI teammates and the level of information abstraction shape what a human supervisor can actually detect. Useful if you are designing dashboards and want evidence on what they hide.

SiliconANGLE (2026). Human-in-the-loop has hit the wall. It's time for AI to oversee AI. An industry read on the scaling limits of human review, and a useful counterpoint: its proposed answer, automated oversight of automation, raises the calibration questions this post argues are still unsolved.

References