RAG — retrieval-augmented generation — is the architecture behind most serious “chat with your documents” systems. When you ask a question, the system first retrieves the most relevant passages from your document corpus, then has a language model compose an answer from those passages. The model answers from what was retrieved, not from whatever it half-remembers from training.
Private RAG is the same architecture run under your control: your documents stay in an environment isolated to your organisation, they’re never used to train shared models, and access follows your permissions. The contrast is with public AI tools, where your uploads leave your control and the terms of use decide what happens next.
Why regulated organisations insist on it
Three reasons come up in almost every conversation we have with compliance-heavy teams:
Data control. A law firm’s matters, a bank’s client files, a trust’s clinical policies — none of these can sit in a consumer chatbot. Private RAG keeps the corpus inside a boundary you can point to in a data protection impact assessment.
Verifiability. Because answers are composed from retrieved passages, a well-built RAG system can cite its sources — the exact clause, page or paragraph. That turns AI from “trust me” into “check me”, which is the difference between a tool compliance will approve and one they’ll ban.
Honest failure. When the documents don’t contain an answer, a grounded system can say so instead of guessing. In regulated work, “the documents don’t say” is a useful answer; a confident fabrication is a liability.
What to evaluate when someone sells you private RAG
Isolation, specifically. “Private” should mean your organisation’s data is separated from every other customer’s — ask how, and whether cross-tenant retrieval is architecturally impossible or just policy.
Citations, at passage level. A link to a whole document is not a citation. You want the exact passage, one click away, inside the reading flow.
Access control that mirrors your structure. Departments and roles should scope who can ask what of which documents. If everyone can query everything, the tool has just dissolved your information barriers.
The messy-document test. Real corpora are scanned PDFs, faxes and forty-year-old precedents. Bring your ugliest documents to the demo — this is where systems separate.
Where it runs. UK hosting, or deployment inside your own environment, should be options rather than negotiations.
Where the knowledge network comes in
Retrieval alone treats your documents as a pile to search. The next layer — the one we’ve built into Marella — is a knowledge network: the people, companies, topics and concepts inside your documents, extracted and linked so the system understands your world rather than just indexing it. Your team curates the ontology; the asset compounds as documents arrive.
That’s the practical distinction to keep in mind: search finds documents, RAG answers questions, and a knowledge network remembers. Regulated organisations generally need all three — privately.
