Where your data goes when staff use AI

A credit analyst is three annual reviews behind. She pastes a borrower's financial statements and the loan officer's notes into a free chatbot and asks for a draft credit memo. Two minutes later the draft is on her screen, and it reads well. The chatbot's terms let the vendor keep what she typed. The tool is missing from the bank's vendor list, the paste is missing from every log the bank keeps, and the only record is her browser history.

That kind of leak tops the list in the Cloud Security Alliance's 2026 survey of 340 people who work on cloud, AI, security, compliance, and risk at financial services firms. Sixty-one percent named leaks of sensitive data through AI use as a top AI security concern, the most common answer. Twenty percent said their firm had a known AI-related security incident, and another 21 percent were unsure. Anjuna Security sponsored the survey.^[1]

How often do staff put firm data into AI tools?

Often, and mostly out of sight. In Microsoft and LinkedIn's 2024 survey of 31,000 knowledge workers, 78 percent of the AI users said they bring their own AI tools to work.^[2] In a 2025 study of 48,000 people by KPMG and the University of Melbourne, 57 percent of employees said they hide their AI use.^[3]

The same study found that almost half of employees admit to using AI in ways that break company policy, such as uploading sensitive company data to free public AI tools.^[3] A written policy has a hard time with habits like these. In a 2026 survey of 2,500 IT and security decision makers by Mimecast, a security vendor, 91 percent of organizations reported trouble getting employees to follow security policy.^[4]

IBM put a price on it. Its 2025 Cost of a Data Breach study, run by the Ponemon Institute, covered 600 organizations that had a breach. One in five reported a breach tied to shadow AI, meaning AI tools used without approval. Organizations with heavy shadow AI use paid about $670,000 more per breach than those with little or none. Breaches involving shadow AI exposed personal data in 65 percent of cases, against 53 percent across the study.^[5]

Where does the data go once it leaves?

It goes to the vendor, under the terms of the account. A personal account runs on consumer terms, which set how long the vendor keeps chats and whether they help train future models. Those terms are a deal between the vendor and the employee, and the vendor can change them.

In August 2025, Anthropic asked users of its Free, Pro, and Max plans to choose whether their chats could help train its models. For users who said yes, it raised retention from 30 days to five years. Its business plans and its API were excluded from the change, so the type of account decided the terms.^[6]

The rules for banks, credit unions, brokers, funds, and advisers all expect a program for the outside firms that handle customer data. The banking agencies' security guidelines, and NCUA's for credit unions, call for due diligence before you pick a service provider, a contract that binds it to safeguards, and monitoring after.^[7][8] Reg S-P, as the SEC amended it in 2024, requires brokers, funds, advisers, and transfer agents to oversee their service providers. Part of that duty is making sure a provider reports a breach within 72 hours.^[9] A personal chatbot account skips every step of that program.

The chatbot tab is the most visible path. Data also moves by other routes:

• AI features in tools you already approved. A vendor turns on an AI summary in the CRM or the document platform, and your data may now pass through a model provider the vendor chose.
• Browser extensions. An AI extension that summarizes pages can read whatever is on screen, including an internal loan system.
• Scripts. A developer calls a public model's API from a script to clean up a spreadsheet, and the traffic skips procurement.
• Connectors. An assistant linked to a shared drive can hand a teller a passage from a file meant for HR. Search that matches meaning skips the permission check unless someone builds it in. Our guide to permission-aware AI covers the fix.
• Agents. An agent that runs on a broad service account can read, and send, far more than the person who started it.

Who can read the model's working notes?

With the leading cloud AI vendors, only the vendor can read them in full. Reasoning models write out working steps before they answer. Anthropic, OpenAI, and Google encrypt those steps and send them to API customers as a sealed block, which the customer's software passes back with the next request. The vendor holds the key.^[10]

That block is a record of your firm's own work, and only the vendor can read it, sort it for retention, or produce it for an examiner. When an examiner asks what an agent weighed before it acted, the record in your hands is ciphertext.

In August 2026, researchers found a way to read the blocks. They fed a sealed block to a weaker model from the same vendor, which wrote it out in plain text. From 315,320 blocks in agent sessions that developers had posted on GitHub and Hugging Face, they recovered 367 pieces of personal data and 182 credentials. In one session, a coding agent asked to strip API keys from a repository before it went public restated the keys inside its hidden reasoning. The vendors fixed the attack after the researchers reported it.^[10]

The same study showed an attacker could hide instructions entirely inside a sealed block, a form of prompt injection that monitors reading the visible conversation will miss. The researchers advise stripping every reasoning block from a transcript before it goes anywhere public.^[10] Add that to the rules for anything your developers post.

What happens when the vendor changes the product?

The data's path changes with it, on the vendor's schedule. A shared cloud AI service ships each release to every customer at once. Your change control governs what your own teams deploy, and the vendor releases on its own calendar. A feature that changes where data goes can arrive overnight inside a tool your staff are already cleared to use.

In July 2025, ChatGPT's share dialog offered a checkbox labeled "Make this chat discoverable," with small print saying the chat could then show up in web searches. Fast Company found thousands of shared conversations in Google results. OpenAI pulled the option that week, saying the experiment gave people too many chances to share things by accident.^[11]

Ask each AI vendor, in writing, how it will tell you about a change to where your data goes. Ask too whether you can turn the change off before it reaches your users. Then ask your own team how long it would take to spot such a change without that notice.

What happens when the provider goes down?

The firms that rely on it can go down together. Concentration risk means many firms rely on the same few providers, so one failure can hit many of them at once. A fault in AWS's Northern Virginia region disrupted services from late on October 19, 2025, until the next afternoon, more than 14 hours.^[12]

Coinbase and Robinhood were among the firms hit, and Robinhood said the cause was a problem at AWS, one of its third-party vendors.^[13] AI adds another layer of the same dependency. The Financial Stability Board put third-party dependencies and service provider concentration first on its November 2024 list of AI weak points that could raise risk across the financial system.^[14] In the CSA survey, third-party risk was the top cloud security concern, named by 55 percent.^[1]

For banks, the 2023 interagency guidance on third-party relationships, FDIC FIL-29-2023, asks for contingency plans in case an activity has to move to another provider or come back in-house.^[15] The guidance covers third-party relationships in general, so it reaches an AI service that drafts credit memos, scaled to how much the bank relies on it. Decide now which workflows stop if the AI service stops, and how staff work without it.

Running the model yourself changes whose outage you plan for. On your own hardware, the AI tool follows your data center's failover plan. In your own cloud account, it follows the plan your other workloads there already have.

How do you find the AI tools staff already use?

Run a 30-day first pass across four sources: network logs, expense records, browser extensions, and an amnesty survey. Pull 30 days of DNS or proxy logs and count traffic to AI domains by department. Have finance search a year of expense lines for AI vendors. Send the survey in the same window.

Managed browsers can list installed extensions, so flag any with AI features that read the page. If you run data loss prevention tools, add large pastes and uploads aimed at AI sites. Those show who moved real content into a chatbot, beyond who visited one. Then check the tools you already approved. Ask each vendor which AI features are switched on in your CRM, document platform, and email, and where those features send data.

Make the survey safe to answer. Promise amnesty for honest answers, and say you are mapping what help people need.

Put the lists side by side. The overlap is your confirmed inventory. Tools that show up in the logs and are missing from the survey tell you where trust is thin. Tools named in the survey and missing from the logs point to personal phones and home laptops handling work data, which is a finding of its own.

Write down each tool, who uses it, and the data it touched. At a bank that means customer NPI, loan files, account statements, and case notes. At a fund, add anything that might be material nonpublic information. A dated inventory with owners and fixes is evidence you can hand an examiner. Run the pass again each quarter. In IBM's study, only 34 percent of organizations with an AI governance policy audit regularly for unsanctioned AI.^[5]

Which controls should come first?

Start with a sanctioned tool people want to use, because staff bring their own AI to get work done. A ban with nothing to replace it pushes the same work further out of sight. Then label your sensitive data, since every rule about what AI may read needs those labels to act on.

Next, control what the tools can reach. Answers and agents should draw only on what the person asking can already open, which means checking permissions at the moment of retrieval. Give each agent its own identity and the narrowest access its task needs. Keep logs you can rebuild an incident from: the prompt, the documents retrieved, the tool calls, and the answer. Those logs turn the 21 percent "unsure" in the CSA survey into a yes or a no.

What changes when the model runs on your own servers?

The path gets shorter, and the records stay yours. Lumen, the private AI platform from Cognetryx, runs on your own servers, so prompts, documents, answers, and logs stay in an environment your team controls. The model runs there too, so its reasoning is produced on your infrastructure.

• Answers follow each person's existing permissions, with sign-in through your SSO over SAML or OIDC.
• Answers cite their sources. Clicking a citation opens the document with the passage highlighted.
• The Compliance Portal logs chats, tool calls, agent runs, approvals, exports, permission and config changes, and sign-ins, and your compliance team can search it.
• Updates ship as signed packages your IT team installs on its own schedule, so a new feature reaches staff when your change control approves it.
• It runs the AI model your firm chooses.

Private deployment shortens the path your data travels. The labels, the permissions, and the quarterly inventory are still your team's work, and our security page has more on how the platform is secured.

Keith Kennedy, CISSP

Founder, Cognetryx

Keith is an IT thought leader with nearly 20 years of experience architecting secure technology solutions for regulated industries. He holds a CISSP certification and has advised enterprise companies on HIPAA, SEC/FINRA, and GDPR compliance.