A BSA officer asks the firm's AI tool how long the bank keeps records of wire transfers. The answer comes back with a citation to the records retention policy. Before that answer goes into a response to an exam request, someone should open the passage and ask four questions. Does it say what the answer says? Is it the current, approved version? Does it apply to this part of the business? Was the person asking allowed to see it?
Tools built around citations still get things wrong. In 2024, researchers at Stanford's RegLab and Human-Centered AI Institute tested the AI research tools that LexisNexis and Thomson Reuters sell to lawyers. The tools searched real legal sources before answering, and they still gave incorrect or unsupported answers on 17 to 33 percent of queries.[1] The study counted an answer as a failure if it was wrong or if it cited a source that didn't back it up.
Bank model risk guidance leaves the design of that check to you. The revised interagency guidance, SR 26-2, issued in April 2026, puts generative and agentic AI outside its scope.[2] For broker-dealers, FINRA's Regulatory Notice 24-09 reminds firms that their existing rules, supervision included, apply when they use generative AI.[3] Either way, the review process for AI answers is one your firm writes and then has to explain to an examiner.
What does a citation prove, and what does it leave open?
A citation shows which document the system used. It leaves open whether the passage supports the answer, whether the document is the current approved version, whether it applies to this business line, and whether the person asking was allowed to see it. A reviewer needs all four answers before the output goes into an exam file.
Take an answer that says a type of record must be kept for a set number of years. The cited policy gives that period for one product line only, and the answer states it for all of them. The citation is real, and the answer is still wrong.
So the check has two layers. First, did the system pull the right document? Second, does the passage back the claim, with its conditions intact? A reviewer needs both, and each one needs its own test.
Why does the order of search and writing matter?
A tool can find the source first and write only from what it found, or it can write first and attach a link afterward. The second design can cite a page that never backs the claim, because the link was chosen to match an answer already written. Ask any vendor which order its system uses.
Lumen, the private AI platform from Cognetryx, finds the source first. It searches your documents, checks what the person asking is allowed to see, and only then lets the model write, working from the passages it found. The model never sees pages outside that person's permissions, because the user's identity is part of the search. Answers follow each person's existing access, with sign-in through SSO over SAML or OIDC.
That order narrows the room for a made-up answer, and it leaves the checking still to do. A model can misread a passage or drop a condition from it, and the next checks are built to catch that.
How do you check that the passage backs the claim?
Read the cited passage next to the answer, one claim at a time. Ask whether the passage states the claim outright, supports it by clear inference, or only covers the same topic. Anything headed to an examiner needs the first kind. Split long answers into single claims so each one gets its own check.
Watch the small words. "May," "generally," "unless," "subject to approval," and "within 30 days" often carry the real meaning of a policy. An AI tool can smooth these conditions into a cleaner answer that sounds more certain than the source. Say the answer reads, "Wires above the limit need a callback." The procedure says a callback is required unless the customer is on the approved list, or the request comes through the secure portal. The answer dropped both exceptions, and your operations team works from the exceptions.
Compound answers need the same care. If an answer says a step is required, applies to every branch, and must be done within a set time, check each of those three claims against the text.
Is the source current, approved, and in scope?
A passage can be accurate and still be the wrong source. A training deck may sum up a policy loosely. A draft may be filed next to the approved version. An old procedure may still be on a shared drive after the board approved a new one. Check approval status, effective date, and scope before relying on it.
Decide in advance which sources can back which uses. The board-approved policy library can support an exam response. A working note can help an analyst start research, and it should stay out of anything headed to a regulator.
Check currency when the answer is written, from the document's own details: effective date, version, approval status, and whether it has been archived. When a policy changes, you should be able to find the saved answers and reports that relied on the old version.
Scope matters as much. A holding company may run a bank, a broker-dealer, and an adviser under one roof, each with its own procedures and its own examiners. Tag each document with the entity, product, and state it covers, so the search can filter out sources that don't apply before the model sees them.
The person asking also has to be allowed to open the source. A citation to a document the user can't access tells them the document exists, and maybe what it says. Our guide to permission-aware AI covers how to test that.
What should happen when no source fits?
The system should say so, name what it searched, and send the question to the person who owns the answer. A clear "no source found" tells the analyst where to look next, and it keeps a guess out of the exam file. Set the rules for declining in advance, and log every refusal.
Set the tool to decline in four cases:
- The sources conflict.
- A source is missing key context.
- No source the person may see covers the question.
- The question calls for a judgment the documents can't support.
Our guide on judgment calls covers that last case.
Log refusals the same way you log answers: the question, what the system searched, and why it stopped. Over a quarter, that record shows where your policy library has gaps. It also shows an examiner that the tool holds back when the documents run out. In Lumen, a refusal is saved like any other chat, in the Compliance Portal your compliance team can search.
What should the record of each answer include?
Enough for someone else to repeat the check months later. Keep the question, the answer, the model version, the passages the system pulled, the version of each source, who asked, and any reviewer's decision. With that record, sampling, incident review, and exam requests become a search of the log.
Each citation should carry stable details: the document name, the system of record that holds it, the document ID, the version or effective date, the page or section, and the exact passage. A citation that reads "Loan Policy, page 14" breaks as soon as a second edition exists.
In Lumen, clicking a citation opens the document with the passage highlighted. The Compliance Portal logs chats, tool calls, agent runs, approvals, exports, permission and config changes, and sign-ins, and your compliance team can search it.
How do you know the checks keep working?
Test the tool like any other control. Build a set of real questions with known right answers, including tricky ones and ones that should get a refusal. Score whether it pulled the approved source, cited the right passage, kept the conditions, and declined when it should. Run the set again after every model, connector, or policy change.
Sort the failures by type. You may find the tool does well on policies and struggles with tables in rate sheets, or pulls old documents from one shared drive. Each finding points to a fix in how documents are loaded, tagged, or filtered.
Match the review to the stakes. Routine lookups, such as a teller checking a procedure, can rest on the citation and the user's own read. Exam responses, board reports, regulatory filings, and letters to customers should get a named reviewer who signs off. Our guide on human review as a control covers how to make that sign-off hold up.
Start with the questions your team answered for the last exam. Run them through the tool, check every citation, and keep what you find. That record is the first thing to show when an examiner asks how you know the AI's answers are right.
Click a citation and read the passage
See Lumen answer from real financial documents, with each citation opening the source at the highlighted passage.
See it in actionSources
- Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, and Daniel E. Ho, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools," Stanford RegLab and HAI, May 30, 2024. Lexis+ AI, Westlaw AI-Assisted Research, and Ask Practical Law AI each gave incorrect or unsupported answers on 17% to 33% of queries. arxiv.org/abs/2405.20362; summary at hai.stanford.edu
- Board of Governors of the Federal Reserve System, OCC, and FDIC, SR 26-2, Revised Guidance on Model Risk Management, April 17, 2026. Generative and agentic AI models are outside its scope. See OCC Bulletin 2026-13. occ.gov
- FINRA, Regulatory Notice 24-09, FINRA Reminds Members of Regulatory Obligations When Using Generative Artificial Intelligence and Large Language Models, June 27, 2024. finra.org
This article is informational and not legal or compliance advice. Confirm how any rule applies to your firm with your own counsel and compliance team.
