When a lawyer pastes a client memo into ChatGPT or uploads a contract to a cloud AI tool, that text leaves the firm's network. It travels to an external server, where a model processes it. Depending on the vendor's data use policies, it may be retained, logged, or used in ways the firm doesn't control and the client didn't consent to. That's the confidentiality problem, and it's architectural.
A private LLM is a language model that runs entirely inside the firm's own environment. Queries go to a server inside the building or inside the firm's private cloud. The model processes the request and returns an answer without any data leaving the network. There's no API call to OpenAI or Anthropic. No query touches third-party infrastructure. The difference matters legally, not just operationally.
The American Bar Association issued Formal Opinion 512 in July 2023, addressing lawyer obligations when using generative AI. The opinion maps existing Model Rules onto AI use: Rule 1.1 (competence), Rule 1.6 (confidentiality), and Rules 5.1 and 5.3 (supervision of lawyers and nonlawyers). Firms must understand how their AI tools handle client information, verify outputs before relying on them, and supervise any AI-assisted work product. The opinion does not ban AI. It requires firms to know what they're using and where client data goes.
What Actually Goes Into a Legal AI Query
Lawyers don't ask abstract questions. They ask things like "Summarize the key risks in this merger agreement" and paste the full document. Or "Draft a demand letter based on these facts" and include client communications, deposition excerpts, financial records. The query itself contains privileged information.
That information has to go somewhere before the AI can process it. In a cloud-based tool, it goes to an external server. In a private LLM, it stays inside the firm's network.
Model Rules 1.6(a) prohibits lawyers from revealing information relating to client representation without consent. A few cloud AI vendors include carve-outs in their enterprise agreements stating they don't use customer data for model training. That's worth something. It doesn't fully address whether data is retained, how long, who at the vendor has access to it, and whether it could be subject to a government request or discovery in litigation. An on-premises model has no such exposure because the data never touches external infrastructure.
The Third-Party Doctrine Problem
There's a less-discussed risk: what voluntary disclosure to a third-party AI vendor might do to privilege claims down the road. Attorney-client privilege protects communications made in confidence between attorney and client. The third-party doctrine holds that when information is voluntarily shared with a third party, privilege may be weakened or lost entirely, depending on jurisdiction and context.
Courts haven't settled how AI vendor access fits this doctrine. But the risk is real enough that privilege counsel in high-stakes litigation is starting to ask whether law firm AI tools created a disclosure event. For a firm that submitted privileged work product or client communications to a cloud AI service, that's a question they'll have to answer.
When the LLM runs inside the firm, there's no third-party disclosure. The only entities with access to the query are the firm's own systems and personnel.
What "Private" Means Architecturally
The term gets used loosely in vendor marketing. Worth being specific: the model weights sit on hardware the firm owns or controls; queries and outputs never leave that network; and the firm controls who can access the system.
Some vendors describe their products as "private" because they use dedicated cloud tenancy or contractually restrict data use. Those are policy commitments. On-premises deployment means the data physically can't leave. The distinction matters when a regulator or opposing counsel asks where client data went.
The hardware requirement is real but manageable. Modern open-weight models suited for legal work run on GPU-accelerated servers that fit in a rack. A firm doesn't need a data center. Smaller firms often work with an MSP who manages the hardware. What they get is a model running in an environment they control, with audit logs they retain.
A 2024 Stanford RegLab study evaluated five AI tools purpose-built for legal research. The error rate ranged from 17% to 33%. That's between one in six and one in three answers containing a significant inaccuracy. General-purpose models performed worse. The implication for private LLM deployment is that grounding the model in the firm's own documents is as important as keeping data in-house. A model that retrieves answers from the firm's verified case files, contracts, and precedent library produces more accurate output than one relying solely on training data.
Use Cases Where Private LLM Actually Fits Legal Work
The practical value is in retrieval and drafting work that happens before attorney judgment gets applied.
Matter research and precedent lookup. Attorneys spend real time finding relevant internal work product: prior briefs, research memos, deposition outlines that touched similar issues. A private LLM grounded in the firm's document library can surface what already exists, with citations, in seconds. Associates stop re-doing research that partners have already done.
Contract review and risk flagging. First-pass contract review against a firm's standard positions is well-suited to this technology. The model flags clauses that deviate from preferred language, missing provisions, or terms that have caused problems in prior matters. Attorneys review the flags; they don't read every line cold.
Document-intensive litigation support. Production review sets can run into hundreds of thousands of documents. A private LLM can pre-screen, group by relevance, and surface the highest-priority items for attorney review. Because the model runs inside the firm's environment, counsel can work through opposing party documents without those documents touching external infrastructure.
Deposition and hearing preparation. Pulling together transcripts, interrogatory responses, and exhibits from prior related matters before a deposition takes hours. A private system grounded in matter files can draft a timeline of key facts, flag inconsistencies across witness statements, or produce a Q&A outline based on the record. That's time attorneys get back.
Internal policy and procedure retrieval. Larger firms have billing guidelines, conflicts procedures, and engagement templates scattered across shared drives. Staff spend time hunting. A private LLM indexed against those documents answers policy questions and links to the source.
Accuracy Is an Architecture Problem
The Stanford research result is worth sitting with. One in three answers wrong is an unusable tool for legal work. But the error rate isn't a fixed property of the technology. It's a function of what the model has access to and how retrieval is structured.
A model that relies on training data to answer legal research questions will hallucinate citations. The cases it cites may not exist, may not say what the model says they say, or may have been overruled. This is a well-documented failure mode. Attorneys have been sanctioned for submitting AI-generated briefs with fabricated citations. The duty of competence under Rule 1.1 means verification isn't optional.
A model grounded in retrieved documents pulls from an indexed library of actual source material and generates answers with citations to specific documents. The model can still be wrong, but it's wrong in verifiable ways. The attorney can see what was cited and check it. That's a supervision workflow that satisfies Rule 5.3.
Private LLM deployments that use retrieval-augmented generation (RAG) against the firm's own case library, brief repository, and research files start closer to the right answer because they're working from verified source material, not generalizing from training data.
What Firms Should Ask Before Deploying
Any firm evaluating an AI tool should be pressing the vendor on a few basic questions before signing anything. Where does query data go during processing? Where is it stored afterward, and for how long? Does the vendor have access to it? Can it appear in any training pipeline, even anonymized? Who at the vendor can access it, and when?
If the vendor answers those questions with contract language rather than architecture, that's worth noting. "We contractually prohibit training on your data" is a policy. "The model runs on your server" is a fact.
Firms should also ask what the model does when it doesn't know the answer. Does it say so, or does it generate confident-sounding text that happens to be wrong? For legal work, a model that admits uncertainty is more useful than one that fills gaps confidently.
Ask about audit logging. ABA Opinion 512 expects firms to supervise AI-assisted work product. That supervision requires records. A private LLM deployment should log what queries were submitted, what documents were retrieved, and what outputs were generated, in a format the firm controls and retains.
The Firm Policy Question
ABA Formal Opinion 512 is explicit that using AI without a governing policy is itself a competence problem. Lawyers are using AI now. Most firms don't yet have a written policy that addresses which tools are permitted, what client data can be submitted to them, how outputs must be verified, and what disclosure obligations exist when AI contributed to work product.
A private LLM deployment narrows what the policy has to cover. The data handling questions are settled by where the model runs. Policy can focus on use standards, verification, and supervision. That's a more tractable problem than trying to govern tools that send client data to external infrastructure with varying and often opaque retention terms.
If a client or regulator asks where their information went, a firm running a private, on-premises LLM has a clean answer: it stayed inside. No vendor policy to interpret, no third-party server to explain.
Built for Confidentiality from the Ground Up
Cognetryx deploys entirely inside your firm's network. Client data stays inside. Queries, outputs, and retrieval logs are captured in an audit trail you control. No third-party API calls. No data egress.
Request a Demo →