Ask an AI assistant whether a transaction is suspicious and you will get an answer. Ask it whether sharing a record with a billing vendor meets the minimum necessary standard and you will get an answer. Both will be fluent, structured, and reasonable. One of those questions might have a knowable answer. The other is a judgment the regulation deliberately assigned to your institution, and the model just made it for you without mentioning that it had.
The output looks identical either way. This is not an accuracy problem, so none of the accuracy controls you have built will catch it.
Regulated rules use judgment language on purpose: reasonable efforts, reason to suspect, material, appropriate. Those words assign a decision to a person. AI resolves them silently and states the result as confidently as it states a fact, and no citation check catches it because nothing in the answer is false. You cannot fix this with a better model. You fix it with a human approval gate before consequential action and a tamper-evident record a reviewer can check afterward.
What is the difference between a hallucination and a manufactured verdict?
A hallucination is a false claim about something that has a knowable answer. The model invents a case citation, misstates a retention period, attributes a rule to the wrong agency. It is wrong, and it is catchable. Open the source, compare the passage, and the error surfaces. That is the failure mode the industry has spent three years learning to detect, and it is the subject of our work on validating AI citations.
A manufactured verdict is different, and it is worse in the ways that matter to a compliance team. Every source is real. Every citation opens. The reasoning is coherent. The answer is still just one defensible position out of several, presented as though it were the position. Nothing is false, so nothing trips. Accuracy testing runs clean over the top of it.
The two failure modes need different controls. Citation validation is the control for the first. It does nothing for the second, because the second is not an error. It is a choice the model made and did not disclose.
Which regulated questions have no single right answer?
More than most teams assume. Regulators write judgment standards into rules deliberately, because a bright-line test would either be gamed or would fail the cases it did not anticipate. The vagueness is the design, and the discretion it creates belongs to the institution.
Two examples carry a great deal of daily traffic in regulated work:
Under HIPAA, a covered entity "must make reasonable efforts to limit protected health information to the minimum necessary to accomplish the intended purpose of the use, disclosure, or request" (45 CFR 164.502(b)). Reasonable is doing the work in that sentence. The rule does not say how much is enough. It says you have to decide, and be able to defend the decision.
Under the Bank Secrecy Act, a bank must file a suspicious activity report when it "knows, suspects, or has reason to suspect" that a transaction meets certain conditions, including that it "has no business or apparent lawful purpose or is not the sort in which the particular customer would normally be expected to engage, and the bank knows of no reasonable explanation for the transaction" (31 CFR 1020.320(a)(2)). Reason to suspect is not a threshold you can compute. It is a call an experienced person makes and documents.
The pattern repeats across materiality, adequacy, appropriateness, and reasonableness tests in nearly every regime. What makes these dangerous in an AI workflow is that they do not look like judgment calls when they arrive:
| The question a user types | What it looks like | What it actually is |
|---|---|---|
| "Can we send this record to the billing vendor?" | A policy lookup | A minimum necessary judgment the covered entity owns |
| "Is this transaction suspicious?" | A classification | A reason-to-suspect call the bank owns and must defend |
| "Is this finding material?" | A threshold check | A weighing exercise with no single defensible number |
| "Does this clause create exposure?" | A contract question | A legal opinion contingent on facts not in the document |
A user asking the left column believes they are asking the middle column. The assistant answers as though the middle column were true. Nobody in the workflow is told that the right column is what actually happened.
Why does AI sound equally confident either way?
Because confidence is a training artifact. Models are tuned against human preference data, and human raters score decisive answers above hedged ones. Decisiveness is what gets reinforced. A model that says "this depends on which standard you apply" tests worse than one that picks, so the picking behavior survives.
There is measurement on this. A 2025 preprint from Junchen Ding and colleagues put 14 leading models through 27 trolley-problem scenarios across ten moral framings, collecting 3,780 binary decisions with written justifications. Two findings are worth carrying into a procurement conversation. Models varied significantly with each other and across ethical frames, so there was no stable answer to find. And the reasoning-enhanced models were more decisive and produced more structured justifications, yet in the authors' words did not always align better with human consensus. The paper also found models were sensitive to ethically irrelevant cues, meaning the framing of a question moved the verdict.
Better reasoning made the answers more convincing. It did not reliably make them more right. Do not assume the next model release solves this.
We demonstrated the same thing publicly with a deliberately silly example: five lobsters on one track, a cat on the other, from Level 7 of Neal Agarwal's Absurd Trolley Problems. The scenario has four defensible answers depending on the frame you apply, and two of them come out of utilitarianism alone depending on whether you count lives or weigh capacity to suffer. Most models pick one and explain it beautifully. The full breakdown is on LinkedIn. Lobsters are a joke. The voice is not, because it is the same voice you get when you ask about a claim denial.
Can a better model fix this?
No, and the reason matters for how you spend your budget. Confidence on subjective questions is not a defect that a larger model corrects. It is a product of how models are trained to be useful, and the evidence above suggests it gets stronger as reasoning improves, not weaker.
This is the same conclusion we reached about accuracy in what zero-hallucination really means. The tools built specifically for legal research still get a substantial share of answers wrong, and better models have not moved that number, because the cause is one layer deeper than the model. Judgment has the same shape. Waiting for a model that knows when to stop being sure is not a plan.
It is also worth checking what your model risk program actually covers. The Federal Reserve issued SR 26-2, Revised Guidance on Model Risk Management, on April 17, 2026, superseding SR 11-7, which had been the governing guidance since 2011. Expectations here are in motion for the first time in fifteen years, and the scope question matters for anyone assuming an existing framework already covers a generative assistant. We cover what changed, and who it applies to, in model risk management for community banks. If your AI assistant was mapped to a framework written before generative systems existed, that mapping deserves a fresh read rather than an assumption.
What should the architecture do instead?
If the model layer cannot be trusted to flag a judgment call, the surrounding system has to make the judgment visible and reviewable. Three properties do most of that work.
- A human gate before anything consequential happens. An assistant that drafts a SAR narrative is useful. An agent that files one is a governance problem. The gate is the point at which a person owns the call the regulation assigned to them, and it needs to be on by default rather than a setting somebody remembers to enable.
- A record that survives the people who made it. The prompt, the response, the evidence retrieved, the user identity, the model version, and the reviewer's decision. A judgment call cannot be reconstructed from its conclusion, because the conclusion never shows the alternatives that were available.
- A log that resists the insider. An audit trail that a privileged user can quietly edit is not evidence. Hash-chaining each record to the one before it makes tampering detectable rather than merely discouraged, and it lets a reviewer verify the chain independently instead of trusting the system that produced it.
Cognetryx is built this way. Agents run under the calling user's credentials with curated tool allowlists, human approval on outbound actions is on by default, and every external tool call produces its own audit record. Security-relevant events are written to a hash-chained log that verifies itself and can be re-verified outside the platform, which defends the trail against someone with write access and not just read access. Retrieval honors your existing access controls before the model sees a document, so the evidence behind an answer is evidence the user was entitled to. All of it runs inside your network, on your infrastructure, with no third-party AI API in the path.
None of that makes a model wise about ethics. It is not supposed to. It makes the model's contribution inspectable and bounded, so a judgment call reaches a person before it reaches a filing.
How do you audit a decision the model never explained?
You mostly cannot, which is the argument for building the record before you need it. An examiner asking why a SAR was not filed in March is asking a question about reasoning, and the reasoning has to exist somewhere durable. Our piece on what BSA examiners test in SAR decisions covers this on the human side: they probe whether your team's reasoning was consistent, traceable, and grounded in your own policy. Adding an AI assistant to that workflow does not lower the bar. It adds a participant whose reasoning also has to be traceable.
Practical questions to put to any vendor, and to your own deployment:
- When the assistant answers a question that involves a judgment standard, what in the output tells the user that it did?
- Can a reviewer see the evidence the model used, at the version it existed in at the time?
- Which actions can an agent take without a person approving them, and is that list on by default or off?
- Can someone with administrative access alter the audit trail without it being detectable?
- Can you reconstruct, six months later, who approved what and on what basis?
If the honest answer to the first question is "nothing," that is worth knowing before an examiner finds it rather than after.
What good looks like
Refusal is not the useful behavior. An assistant that declines every hard question is a worse tool than a search box. What helps is an answer that reports its own footing: here is what the policy says, here is where the standard requires a judgment, here are the considerations that pull in each direction, and here is the person who owns the call.
That is a smaller claim than most AI marketing makes, and it is the one that holds up. Your analysts already work this way. A good one does not answer "is this suspicious?" with a verdict. They answer with what they saw, what it resembles, what argues against it, and what they concluded and why. The conclusion is the least interesting part of the response, because the reasoning is the part that gets examined.
Hold the software to the standard you already hold the people to. "It depends, and here is exactly what it depends on" is not the model failing to answer. It is the model telling you the truth about the question.
See what governed AI decisions look like
A short assessment shows how a private deployment keeps consequential actions behind a human gate and every answer inside a record you can verify, in your own network.
Book a Free AI Strategy Assessment →Frequently asked questions
Can AI make judgment calls in regulated work?
It can produce one, which is not the same as being allowed to make one. Rules like HIPAA's minimum necessary standard and the Bank Secrecy Act's suspicious activity standard are written with judgment language on purpose, and they assign that judgment to the institution. A model that resolves the question silently has taken a decision the regulation reserved for a person. The model is allowed to inform the call. It is not allowed to be the call.
What is the difference between an AI hallucination and a manufactured verdict?
A hallucination is a false claim about something with a knowable answer, and a citation check catches it. A manufactured verdict is a confident ruling on a question that has no single correct answer. The sources are real, the reasoning is sound, and the answer is still only one defensible position out of several. No accuracy benchmark catches it, because nothing in it is false.
Why do AI models sound confident about subjective questions?
Because confidence is rewarded in training. Human raters score decisive answers above hedged ones, so decisiveness is what gets reinforced. A 2025 preprint from Ding and colleagues tested 14 models across 27 trolley scenarios and 3,780 decisions and found that models with stronger reasoning were more decisive without always aligning better with human consensus. Fluency is not evidence that the question had an answer.
How do you audit an AI decision that involved judgment?
You need three things the model cannot give you on its own: a record of what it was asked and what it said, the evidence it used, and proof that a person reviewed it before anything happened. That means a human approval gate before consequential action and a tamper-evident log an auditor can verify independently. Reconstructing a judgment call from the final answer alone is not possible, because the answer never shows the alternatives.
