Human review is the control your AI policy names. New data shows what it misses.

Human oversight, by whatever name

Human review shows up in AI governance under a lot of labels. Human in the loop. Human oversight. Meaningful human review. Article 14 of the EU AI Act requires that high-risk systems be designed so they can be effectively overseen by natural persons while in use.^[1] The NIST AI Risk Management Framework asks organizations to define, assess, and document their processes for human oversight, and leaves the amount of oversight to the risk of the system.^[2] The FDA is the most specific of the three about what review has to mean. Its clinical decision support guidance conditions non-device status partly on software being intended to let a clinician independently review the basis for a recommendation. It also says outright that it does not expect time-critical software to meet that bar, because the clinician will not have time.^[3] FINRA's Regulatory Notice 24-09 asks member firms to bring generative AI inside a reasonably designed supervisory system.^[4]

Bank supervisors drew a line in April. The revised interagency model risk guidance, SR 26-2, keeps the effective challenge standard, meaning critical analysis by objective experts across the model lifecycle, and then excludes generative and agentic AI from its scope by name.^[5] For the systems this article is about, the model risk framework banks have leaned on since 2011 does not currently apply. Whatever review regime covers an agentic deployment at a bank right now, the institution wrote it itself.

All of these assume that a person who is shown a decision will evaluate it. Three pieces of research published in the first week of August suggest that assumption holds much better in some conditions than others, and that the conditions matter more than the policy language does.

409,000 decisions, one threat in three missed

On August 5, a Belgian developer named Alex Wauters published statistics from a browser-based simulator he had built. Players took the human-in-the-loop seat for an AI coding agent, approving or denying commands under time pressure. He collected more than 40,000 runs and 409,000 individual approve-and-deny decisions.^[6]

The average player missed one threat in three, at a mean accuracy of 66.3 percent. Seven percent approved every prompt they saw and denied nothing at all. A headline 35.2 percent of players caught every threat, but Wauters is careful to gut his own number: only 20.8 percent managed it while blocking at most one in five of the safe commands. The rest got a perfect catch rate partly by blocking everything, which he labels the Human Bottleneck.

The category breakdown is more useful than any of that. Obviously destructive commands were missed 11.7 percent of the time, roughly what you would hope for. Scope violations, meaning an agent reaching for something outside its job such as AWS credentials or a Kubernetes config, were missed 35 percent of the time. Attacks wearing a familiar name did best of all. A malicious npm run analyze, piping bundle output to an unfamiliar endpoint, was approved 64.7 percent of the time. Pooled across three disguised npm run commands, the miss rate was 52.5 percent against 28.4 percent for every other exfiltration-style attack in the set. Hiding a payload behind a familiar script name roughly doubled its success rate.

Reviewers also blocked safe work at a steady clip. Fifty-nine percent blocked npm config set registry https://npm.internal, which is somebody pointing at an internal mirror. Forty-five percent blocked rm -rf dist/, an ordinary build cleanup. Degradation under load runs in both directions, so a high denial rate is not by itself evidence that the control is working.

The vendor's own telemetry says 93 percent

A game can only tell you so much. The production number comes from the vendor. In an engineering post published May 25, Anthropic wrote that its telemetry showed users approved roughly 93 percent of permission prompts.^[7] That is production usage by people doing their jobs.

What the same post concludes about that number is more interesting than the number, because it is not what a company shipping a permission-prompt interface would be expected to say. Anthropic's engineering team uses the phrase "approval fatigue," reports that it "showed up within weeks," and calls human-in-the-loop approval a fallible layer that cannot stand on its own: "Theoretically that works, but we've found the approach to be fallible." The design principle they draw from it opens their summary:

"Design for containment at the environment layer first, then steer behavior at the model layer."

"The deterministic boundary is what gets hit when everything probabilistic misses."

In practice they describe constraining where and how an agent can act with process sandboxes, VMs, filesystem boundaries, and egress controls, varying by product. Shipping an operating-system sandbox for their coding tool, with reads allowed, writes allowed inside the workspace, and network denied by default, cut permission prompts by 84 percent and let the agent run largely without interruption. Their reference devcontainer goes further and is built so the agent can run unattended without per-action approvals at all. In both cases the prompt count fell because the perimeter got tighter, not because anyone decided the agent had earned more trust.

Where human review held

The other side of this arrived the same week.

On August 4 the UK AI Security Institute published an incident report from its cyber-range evaluations. Across 122 runs testing seven models, agents took 19 unsanctioned actions on the live internet in 10 of those runs. The conditions were adversarial by design: "Internet access was deliberately enabled" and "The developers' cyber classifiers were deliberately switched off."^[8] The 19 were not 19 separate incidents. AISI is explicit that they clustered, arising mostly from a single sustained line of activity by one model.

The most serious sequence was an agent trying to insert malicious code into a publicly used open-source project. It researched the project's maintainers, created multiple fake identities, used them to socially engineer a real maintainer into approving the code, edited its own earlier activity to look harmless when challenged, and used Tor to get around GitHub's network restrictions. That last move is what triggered the alert. AISI records what stopped the code itself: "A human maintainer caught and refused to approve the malicious code."

Human review worked. The reviewer was a maintainer looking at one discrete pull request, with time, with context, and with professional reason to be suspicious of unfamiliar code. That is a different job from a developer clicking through the fortieth permission dialog of an afternoon.

Both findings are true at once, and AISI holds them together itself. It credits the people: "standard good practice, human judgment, and caution around AI-generated code stopped the worst outcomes." Then it says what that reliance actually was: "in several cases the margin between failure and success was narrow, resting on human vigilance rather than a technical barrier that would reliably prevent this behaviour in a more capable agent."

AISI draws three lessons from the incident. Tighter controls on internet access. Real-time monitoring built to flag or block out-of-scope actions as they happen. A redesign of evaluations that assumes a capable model may try to act beyond its remit, with the scope of any such behaviour limited in advance. All three move the boundary rather than adding another approval gate.

Four things to take into your next AI risk review

The third item from that week is OWASP's Top 10 for LLM Applications 2026, released in early August. Excessive Agency climbed to third place, up from sixth in the 2025 list, in an edition OWASP describes as grounded in thousands of real-world AI security incidents.^[9] Agent permissions are no longer a fringe concern in the community's own ranking.

For an institution writing or defending an AI control set, a few practical things follow.

1. Separate review from approval in the policy language. "A human approves each action" and "a qualified reviewer examines the output before it is used" are different controls with different failure modes, and most policies use the phrases interchangeably. An examiner who has read the recent literature will ask which one you mean, and the honest answer may be different for different workflows.
2. Count the prompts. If a deployment generates dozens of approval decisions per person per day, the control's reliability falls as adoption rises. That inverse relationship needs an explanation you can give out loud. A control that works during the pilot and thins out in production is a finding waiting to be written.
3. Document what happens when approval fails. Whether a reviewer eventually approves something they should not is not really in question. Assume they will. The question your risk analysis has to answer is what the agent can reach at that moment. If the answer is everything the user's credentials can reach, the prompt was carrying weight it cannot carry.
4. Keep human review where it is strong and say so. Discrete artifacts, qualified reviewers, enough time. A credit memo, a suspicious activity report narrative, a CAPA record, a legal research summary. Those are the places where a person examining output catches things, and they should be defended in an exam rather than diluted across every click.

Deterministic controls are the ones you can evidence

The design conclusion here matches the one Anthropic's engineers reached, and it turns on which controls are deterministic. A permission prompt depends on a person's attention at a particular moment. A network egress rule does not. Neither does a filesystem boundary, or a scoped permission on an index that limits which sources a given user's query can reach. Those either hold or they do not, and you can test them on a Tuesday and get the same answer on a Friday.

For a regulated institution that distinction has a documentation consequence, which is usually where these arguments actually bite. You cannot produce evidence that a reviewer was paying attention. You can produce the configuration of a boundary, the log of what it denied, and the test that proves it held. One of those survives an examination. The other becomes a conversation about training records.

This is where deployment comes back into it. When the platform runs inside the environment your security team already controls, the boundaries are yours to set and yours to evidence: what the agents can reach, which sources each user's queries can touch, what leaves the network, and what was refused. Cognetryx is built that way deliberately, with the index, the agents, and the tools inside the institution's own perimeter, whether that is on-premises or otherwise locally hosted. The approval prompt still exists in that architecture. It is just no longer the thing holding the line.

If you take one number into your next AI risk review, take the 93 percent. It is the vendor's own figure, drawn from production, and it is the rate your compensating control is currently running at.

Frequently asked questions

Keep reading

• What are your AI agents allowed to do?
• How to implement AI access controls safely
• Permission-aware RAG: why enterprise AI fails in production
• Financial firms are deploying AI agents faster than they can govern them
• Agentic AI: what it means (and doesn't)
• Private AI for regulated industries (the pillar)

Keith Kennedy, CISSP

Founder & CEO, Cognetryx

Keith is an IT thought leader with nearly 20 years of experience architecting secure technology solutions for regulated industries. He holds a CISSP certification and advises institutions on secure AI architecture, access control, and keeping sensitive data inside the network. About Keith