The default assumption in most AI vendor conversations is that cloud is the delivery model and the institution's job is to assess which cloud vendor's security posture is acceptable. That framing serves the cloud vendors well. It doesn't reflect what the technology actually requires, or what the regulatory guidance says about third-party risk.
Banks and credit unions have been running mission-critical applications on-premises for decades. Core banking, loan origination, fraud detection, document management, all of it ran on hardware inside the institution's network before any of it moved to managed hosting or cloud. Today's AI models are large, but financial institutions have been managing compute-intensive infrastructure locally for years.
Yes, banking AI without public cloud is possible, production-ready, and in use at financial institutions today. The hardware requirements are real but manageable. The regulatory posture is cleaner than cloud alternatives under FDIC FIL-29-2024 and the interagency guidance on third-party risk. The cost structure is predictable in a way that token-based cloud pricing isn't. The real question is whether your institution has evaluated it alongside cloud options, or has just been assuming cloud is the only path.
What the Hardware Actually Requires
Modern large language models run on GPU-accelerated servers. The specific hardware depends on the size of the model and the query volume the institution needs to support. A community bank or mid-size credit union with a few hundred users accessing AI for document retrieval, policy lookup, and compliance review is not running the same infrastructure as a hyperscaler. The compute requirement scales with use case and user count, not with some abstract notion of enterprise AI complexity.
Purpose-built AI appliances are available from multiple hardware manufacturers. A single server with appropriate GPU capacity can run a production-grade language model capable of supporting the AI workflows most financial institutions are actually trying to enable: knowledge retrieval, document summarization, compliance question answering, and procedure lookup. That server sits in the institution's existing data center, connects to the institution's existing network, and is managed by the institution's existing IT staff or MSP.
The capital cost is a one-time expense rather than an ongoing per-token or per-seat charge. For institutions that model technology costs on a three-to-five year horizon, the total cost of ownership comparison between owned hardware and cloud AI subscription is often closer than the initial sticker price suggests, and frequently favors on-premises once query volume reaches meaningful scale.
What FIL-29-2024 Actually Requires of Cloud AI Relationships
FDIC Financial Institution Letter FIL-29-2024, along with the interagency guidance on third-party relationships issued in 2023 by the OCC, FDIC, and Federal Reserve, establishes that financial institutions are responsible for managing the risks posed by their third-party relationships throughout the relationship lifecycle. That responsibility does not diminish because the third party is a technology vendor rather than a service provider, and it does not diminish because the service is delivered via API rather than a signed contract for a discrete deliverable.
Cloud AI vendors are third parties under this framework. Using a cloud AI API for banking workflows creates a third-party risk management obligation that includes due diligence on the vendor's security controls, contract terms addressing data handling, ongoing monitoring, and contingency planning. Regulatory examiners are beginning to ask institutions to demonstrate that their AI vendor relationships have been through the same third-party risk management process as any other significant vendor relationship.
The question examiners are asking is not whether you use AI. It is whether your use of AI is inside your risk management framework. Cloud AI used without documented vendor due diligence, without contract terms addressing data handling and breach notification, and without ongoing monitoring is outside the framework. On-premises AI, operated by the institution's own staff under the institution's own controls, is inside it by design.
Data Sovereignty Is Not a Theoretical Concern
When a bank employee queries a cloud AI system, the query travels to an external server. The response travels back. What happens to the query data in transit, at rest on the vendor's infrastructure, and in the vendor's training and model improvement processes depends on the vendor's terms of service and data use policies - not on the institution's information security program.
For a query about a customer's loan status, account history, or credit profile, this is nonpublic personal information under GLBA. For a query that includes draft regulatory correspondence, internal policy details, or strategic planning documents, the data may be sensitive for additional reasons. The institution sending these queries to a cloud AI system has made a decision about third-party data sharing that its information security and compliance teams may not have fully evaluated.
Data sovereignty is a simple concept: the institution controls where its data goes. On-premises AI makes that architectural rather than contractual. Customer data, internal documents, and employee queries don't leave the institution's network. There's no vendor data use policy to evaluate because there's no vendor receiving the data. The institution's own security controls apply, the same ones its examiners already review.
Every cloud AI vendor relationship a financial institution creates is a new third-party risk management obligation. Due diligence, contract review, ongoing monitoring, incident response planning - the interagency guidance requires all of it. On-premises AI eliminates the category. There is no vendor. There is no third-party risk management requirement for a system the institution operates itself. For institutions managing lean compliance teams, that elimination has operational value beyond the security benefit.
The Cost Structure Comparison
Cloud AI pricing is primarily consumption-based. Vendors charge per token, per query, or per seat, depending on the product. For light, occasional use, consumption-based pricing is economical. For institutions where AI becomes embedded in daily workflows - where loan officers, compliance staff, and branch managers are querying the system dozens of times per day - consumption costs accumulate in ways that are difficult to forecast and difficult to control.
On-premises AI has a different cost structure. The capital expense for hardware is a known quantity. Ongoing costs are power, cooling, and maintenance - predictable line items that fit standard IT budgeting. There is no per-query charge. A compliance team that runs five hundred queries in a month costs the same as one that runs five thousand. That predictability has real value for institutions that are trying to scale AI use rather than ration it.
| Factor | Cloud AI | On-Premises AI |
|---|---|---|
| Upfront cost | Low (subscription or pay-per-use) | Hardware capital expense, one-time |
| Cost at scale | Grows with query volume, unpredictably | Fixed regardless of usage volume |
| Data leaves network | Yes, every query | No, never |
| Third-party risk mgmt required | Yes, full vendor lifecycle | No, institution-operated |
| Exam posture under FIL-29-2024 | Vendor relationship in scope | No third-party relationship to examine |
| Model grounded in your content | Requires custom setup, ongoing cost | Deployed against your documents from day one |
| Availability dependency | Vendor uptime SLA | Institution's own infrastructure |
What On-Premises AI Can and Cannot Do
On-premises AI is well suited to retrieval-augmented generation: giving institution staff the ability to ask questions and get answers grounded in the institution's own documents. Policy manuals, compliance procedures, loan guidelines, exam preparation materials, product documentation, and regulatory guidance all become queryable in natural language. The system retrieves relevant source content, synthesizes a response, and cites the source. Staff get the answer and can verify it.
What on-premises AI does not do is provide real-time access to internet content, third-party databases, or live market data feeds. For use cases that require those connections, the architecture gets more complex and the data handling questions multiply. For the use cases that generate the most daily value in a banking environment - answering staff questions against internal documentation - on-premises AI is not a compromise. It is the better deployment for the task.
The Objection Worth Addressing Directly
The most common pushback against on-premises AI is that community banks and credit unions do not have the IT staff to operate it. This is a reasonable concern and worth taking seriously. The honest answer is that modern AI appliances are designed for managed deployment. Initial configuration, model deployment, and document ingestion are implementation-phase activities. Ongoing operation is not meaningfully different from managing any other server appliance the institution already runs.
Institutions that work with managed service providers have an additional path: the MSP manages the AI appliance as part of its existing managed services relationship. The institution gets the data sovereignty and regulatory benefits of on-premises deployment without requiring internal staff to develop new competencies. The MSP channel is one reason on-premises AI is accessible to institutions well below the asset size that most cloud AI vendors target as their primary market.
How to Evaluate Whether It Is Right for Your Institution
The evaluation starts with use cases, not infrastructure. What are the highest-value problems AI would solve for your staff today? If the answers cluster around knowledge retrieval, compliance question answering, document review, and policy lookup, on-premises deployment is well suited and worth a detailed assessment. If the answers require real-time data integration or public internet access, the architecture conversation is different.
From use cases, the evaluation moves to data. What documents would the AI need to know? How current does that knowledge need to be? How sensitive is the content? For most institutions, the answers confirm that the data involved is exactly the kind that shouldn't leave the network. At that point, on-premises stops being a limitation and becomes the right architecture for the task.
The final step is a total cost of ownership comparison across a realistic time horizon. Three years is the minimum useful window. Include hardware, implementation, and ongoing support costs for on-premises. Include subscription, per-query, and implementation costs for cloud alternatives. The gap between the two is frequently smaller than initial conversations suggest, and the regulatory and security benefits on the on-premises side are not reflected in any vendor's pricing comparison.
See the on-premises option clearly before you commit to cloud.
Most institutions evaluate cloud AI first because that is what vendors present. We build the other conversation. Our free AI Strategy Assessment walks through your use cases, your regulatory posture, and a realistic total cost of ownership comparison so you are choosing based on a complete picture.
Book a Free AI Strategy Assessment →