- Analysts get answers that cite the exact policy clause or case note they came from
- Retrieval respects existing entitlements, so the assistant never shows what a user could not already open
- An evaluation set owned by compliance gates every prompt, model and index change
- A full audit trail of questions, sources and answers for second-line review
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A bank's onboarding analysts spent much of each case looking things up. Policy lived across hundreds of pages of procedures, risk appetite statements and local addenda, and the most useful guidance was often in past case notes: how a similar ownership structure or jurisdiction had been handled before. Senior analysts carried that knowledge in their heads; newer analysts asked them, or searched a document portal that matched keywords and returned whole documents.
Customer due diligence obligations come from the FATF Recommendations and their national implementations, which require institutions to identify and verify customers and beneficial owners1. None of that changes because an assistant exists. The aim was narrower and more useful: help analysts find the right rule and the right precedent faster, with the evidence in front of them.
What had to be true
- Answers come from approved sources only, and every answer cites them.
- Entitlements are enforced at retrieval: a user only ever sees passages they could already open in the source system.
- The analyst decides. The assistant informs a decision; it never records one.
- Quality is measured before release, against questions compliance agrees are representative.
- Everything is auditable: question, retrieved sources, model version and answer.
Architecture
The pattern is retrieval-augmented generation: a retriever selects relevant passages and a generator answers conditioned on them, which keeps knowledge in an index that can be updated without retraining2. The work was in making each layer governable.
The data flow, end to end
- IngestOnly published policy versions and closed case notes, with owner and effective date
- IndexSplit on clause boundaries; attach source permissions to every chunk
- RetrieveFilter by the analyst's entitlements, then hybrid search and re-rank
- AnswerGenerate only from retrieved passages, or say the sources do not cover it
- CiteLink each statement to a clause or case; flag unsupported sentences
- LearnAnalyst feedback becomes candidate evaluation cases for compliance review
Hybrid retrieval mattered because KYC questions mix exact terms (a form name, a jurisdiction, a document type) with paraphrase. Keyword search finds the former and vectors the latter. The detail is in hybrid search for RAG and choosing embedding models.
The hard part: permissions that survive retrieval
The tempting shortcut is one index for everyone and an instruction to the model not to reveal restricted material. That is not access control. OWASP's Top 10 for LLM applications lists sensitive information disclosure and vector and embedding weaknesses as distinct risks for exactly this kind of system3. The approach was:
- Carry the source system's permissions onto every chunk at index time, and re-sync them when they change.
- Apply the entitlement filter inside the retrieval query, so restricted passages are never ranked, let alone sent to the model.
- Treat case notes as more sensitive than policy by default, with access scoped by business line and region.
- Test it: evaluation cases that ask, as a restricted user, for content they should not see, and fail the release if anything leaks.
The same thinking applies to retrieved text as untrusted input; see prompt injection defence.
If a user could not open the document, the model should never read it on their behalf. Enforce that in the query, not the prompt.
How it ran
- Asks in plain language
- Checks cited clauses
- Records own decision
- Filters by entitlement
- Retrieves and answers
- Logs sources and version
- Owns the evaluation set
- Approves releases
- Samples the audit log
- Top analyst questions
- Source inventory and owners
- Risk and control mapping
- Evaluation set drafted
- Compliance sign-off
- Baseline retrieval quality
- Entitlement-aware index
- Citation checking
- Audit and tracing
- One team, then more
- Feedback loop into evals
- Runbooks and ownership
Framing the work against the NIST AI RMF's govern, map, measure and manage functions gave risk and compliance teams a familiar structure for their review4. The release checklist followed RAG to production, and the evaluation design followed evaluation sets that predict production.
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Entitlement filter in the retrieval query | Prompt-level instructions | Restricted content never reaches the model; testable |
| Hybrid keyword and vector search | Vector-only retrieval | KYC questions mix exact terms with paraphrase |
| Cite or decline | Best-effort answers | An uncited answer is not usable as evidence |
| Compliance-owned evaluation set | Engineering-owned benchmarks | The people accountable for the outcome define the bar |
| Pinned model versions | Always-latest model | Changes are deliberate, evaluated and logged |
What I'd do again, and differently
Again: write the evaluation set before the first prompt, and put entitlements in the data path from day one. Differently: start the source clean-up sooner. Superseded policy versions and inconsistent case notes caused more wrong answers than the model did, and fixing content is slower than fixing code.
If you are weighing something similar, the AI architect page describes how I run these engagements, or analyse your challenge and bring the result to a call.
- 1The FATF RecommendationsFinancial Action Task Force
- 2Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksarXiv (Lewis et al., NeurIPS 2020)
- 3OWASP Top 10 for LLM ApplicationsOWASP GenAI Security Project
- 4