The problem
For about eighteen months I was the engineer accountable for a RAG platform inside SAP Labs India. It served 400+ internal users across German and Indian sites, sat on a corpus that crossed two million documents, and had to answer under a p95 of two seconds while never leaking personal data across tenant boundaries or across the German/EU data-residency line. It was demoed at DKOM 2025 as one of the internal AI demo pods, and by then the interesting parts of the system had almost nothing to do with the LLM.
I've written three deep-dives on individual components already: the HNSW vs IVF-PQ decision at 2M vectors, the redact-at-retrieval PII architecture, and why I fine-tuned DeBERTa over XLM-R for German PII. This post is the companion piece: how those pieces actually fit into one system, what the boundaries between them cost, and the one production incident that reshaped the retrieval layer more than any offline benchmark did.
Prior art
Before writing a line of code I did a real survey. The interesting axis was not "what retrieves best" (most of these systems retrieve fine). It was posture: what does each of them assume about where data lives, who touches it, and what evidence is left behind.
LangChain / LlamaIndex. The default monolithic RAG stacks. Both are excellent for prototyping and both make the exact tradeoff we could not make: they assume the pipeline runs in one process, sharing one memory space, with PII and non-PII flowing through the same objects. There is no natural place to insert a redaction pass whose outputs are the only thing the generator ever sees, and the audit trail is whatever you bolt on. Fine for a demo. Not fine for a system that has to explain, per query, why a specific token appeared in the response.
Vertex AI RAG (Google). Clean managed offering with strong retrieval. The blocker is a hard one: the corpus lives on Google infrastructure. For a workload where a non-trivial share of documents are subject to BDSG and internal SAP data-classification rules that forbid egress, this was disqualifying before the first benchmark ran. Not a criticism of the product; it just wasn't the product we could use.
Azure AI Search + Azure OpenAI. Closer to viable because SAP has an established Azure footprint and EU regions are available. Two things pushed us off it. First, the PII detector we needed for German legal and HR text was a fine-tuned DeBERTa checkpoint, and getting it inside the Azure pipeline meant either running it as a sidecar (defeats the point) or accepting the built-in classifier (recall on German declined proper nouns was not defensible). Second, per-tenant latency observability was harder to get out of the managed layer than out of a system I owned end-to-end.
Cohere RAG / Command-R stack. Genuinely good at multilingual retrieval; Cohere's embed model was competitive on German. But the same posture problem: for embeddings alone we could arrange EU-region isolation; for the audit trail and the redaction guarantees we could not.
Weaviate / Qdrant self-hosted. These are what we ended up cannibalising ideas from. Qdrant's payload filter design in particular is close to what we needed. The reason we didn't just adopt Qdrant wholesale was integration cost: the retrieval index is one component of six, and the six had to speak the same tenancy and audit protocol. It was cheaper to own an HNSW index we understood than to make Qdrant's filter semantics and our audit envelope agree on edge cases.
FAISS. Meta's FAISS library is the ANN reference implementation, the yardstick every other index gets measured against. In January 2025, if the question had been "retrieve fast on Python objects," it would have been the answer. It was not the question. FAISS is a library, not a database: persistence is serialize-and-reload, there is no transaction story to speak of, and multi-tenancy or access control has to be written in the layer above. Stock IndexHNSWFlat filters after the traversal rather than during it, which is the trap I describe in the burst-filter incident below. Choosing FAISS meant building most of a database around it in application code, and that was the project I was quietly running from.
ChromaDB. At its January 2025 state Chroma was the easiest open-source vector store to stand up: embedded mode with a persistent SQLite backend, a small Python API, a demo you could show a stakeholder in an afternoon. It was a fine choice for a prototype and a poor one for two million documents across four hundred users under BDSG. Chroma was single-node in the deployment shape we would have used; RBAC lived in the application rather than the database; encryption at rest was whatever the disk gave you; the audit trail had to be assembled outside the store. The Datenschutz-Folgenabschätzung would not have cleared any of those.
SAP HANA vector engine. The internal-politics candidate, and the one that turned out to be a serious answer once I sat with it. The HANA Cloud vector engine had been GA since mid-2024, and by January 2025 the certifications relevant to us had caught up: BSI C5 and ISO 27001, EU regional deployment, HDI-container tenancy, encryption at rest, an audit log that was first-class rather than bolted on. Filter-during-traversal came for free through SQL predicates on the same table as the vector column. Because the corpus was already in SAP systems, keeping retrieval in-family avoided a fresh data-egress review and a new vendor procurement cycle. The reason I did not adopt HANA wholesale for the retrieval index sits in the section below: at January 2025 the vector-engine ergonomics inside HANA assumed you would trust the query plan, and I wanted a debugger on the code path that handled a filter selective enough to break normal graph traversal (the specific failure mode described in "What broke first"). For future workloads with the same posture and less need for that level of index-level control, HANA is where I would start.
What we did differently
Three decisions distinguish this system from the prior art above.
Redaction lives downstream of retrieval, not at ingest. The naive GDPR-compliant RAG scrubs its corpus at index time and calls it done. That destroys recall on every query that legitimately mentions a public entity (a board member, a listed customer, a public regulator), because the corpus no longer contains the string the user typed. We keep the corpus intact, retrieve on the full-fidelity text, and run the PII pass between retrieval and generation. Full argument in the redact-at-retrieval post.
Retrieval is HNSW with a metadata pre-filter, not IVF-PQ. The recall floor the reranker needed was above what product quantization gave us on multilingual text. We paid the RAM cost for float vectors. Full argument in the HNSW deep-dive.
Everything a user sees is signed. Every response leaving the platform carries a response signature and an audit envelope pointing to the exact document IDs, embedding model version, redaction model version, and generator prompt hash that produced it. This is not for the user. It is for the compliance review that showed up a quarter later.
The rest of this post is the shape of the system those three decisions produced.
System design
Left-to-right, this is what a query walked through.
1. Query preprocessor
What. Language detection, minor normalization (Unicode NFKC, whitespace collapse, mojibake repair), and tenancy resolution: turning the caller's auth context into a set of allowed document tags.
Why. Two of the six languages in the corpus were mixed-script by default (German with English technical terms, English with occasional Devanagari names). Normalizing here rather than downstream avoided a class of bugs where the embedding model and the reranker saw slightly different strings. Tenancy resolution has to happen before retrieval, not inside it, because it's the input to the pre-filter that runs at HNSW query time.
Number. Preprocessor budget was around 15 ms of the 2s budget. Cheap.
2. Embedding generation
What. A multilingual sentence-embedding model producing ~768-dim vectors. We used a multilingual encoder that had been evaluated internally on German legal + English engineering text; the specific checkpoint is not the interesting variable here. The interesting variable was that it was one model for the whole corpus, not a per-language ensemble.
Why. A per-language ensemble is tempting on paper but forces a routing decision at query time that either (a) doubles the retrieval fan-out or (b) risks a language misclassification pushing an English query at a German-only index. One shared multilingual space made the pre-filter the language axis rather than the model choice.
Number. Embedding was around 20–30 ms on the shared GPU pool, batched at the query-frontier level.
3. HNSW query with metadata pre-filter
What. A single HNSW index over the full corpus with per-vector payload (tenant ID, language tag, document classification, ingestion date, source system). The query supplied k, ef_search, and a filter predicate. Filter evaluation happened during graph traversal, not after.
Why. Post-filtering is a trap at this size. If a tenant owns 3% of the corpus and you post-filter, you have to over-fetch by ~30× to have any hope of k survivors, and the graph traversal cost balloons. Pre-filter during traversal is the only version that stays inside a 120 ms retrieval budget at 2M vectors. Full geometry argument in the HNSW post.
Number. Retrieval slice of the latency budget: ~120 ms p95 at 2M docs. That's the number the whole system was designed backwards from.
4. 8-way parallel retrieval fan-out
What. For each user query the frontier issued up to 8 parallel retrievals: the original query plus up to 7 rewrites (HyDE-style hypothetical answer, translated variants, keyword-heavy variant, etc.). Results were merged, deduplicated on document ID, and score-fused.
Why. Single-query retrieval on multilingual text hit a recall ceiling we could not lift with ef_search alone. Fanning out the query is cheaper than fanning out the index; the index is the expensive artifact. 8 was empirical: 4 helped, 8 helped more, 16 was in the noise and cost real budget. This is the "directionally 8" I'll defend; the exact number moved cycle-to-cycle.
Number. Fan-out added ~40 ms wall-clock, not 8×, because the retrievals were CPU-parallel and shared the vector cache on hot embeddings.
5. Context assembly
What. Reranker (cross-encoder) plus context packing. The top ~50 candidates from the fan-out were reranked; the top ~8 were assembled into a context window with metadata headers per chunk.
Why. The metadata headers matter more than they look. Each chunk carried its document ID and classification into the generator's prompt so that the response could be attributed at citation time. Without them, the audit trail at the end of the pipeline has to reconstruct provenance from string matching, which is exactly the kind of load-bearing string match that fails on refactor.
6. PII detector (fine-tuned DeBERTa)
What. A DeBERTa-v3-base checkpoint fine-tuned on a curated German + English PII dataset, run on the assembled context immediately before it reached the generator. The detector replaced entities with typed placeholders (⟨PERSON_1⟩, ⟨IBAN_1⟩, etc.).
Why DeBERTa over XLM-R. DeBERTa's disentangled attention gave measurably better recall on German declined proper nouns (the case where a name changes form depending on grammatical role), which was the largest single class of misses XLM-R made on our eval. Full argument in the DeBERTa-over-XLM-R post.
Number. ~90 ms p95 on the shared GPU. This was the second-biggest single-component slice after the generator itself.
7. Generator (LLM)
What. An internally hosted LLM behind a small prompt-hardening layer. The generator saw only the redacted context and the redacted query.
Why. The generator was the only stage that did not directly touch PII. That's deliberate. It let us reason about the generator as a stateless function of "redacted context + redacted query → redacted response" and give the compliance reviewer a single sentence they could hold onto.
8. Response signing and audit envelope
What. Every outbound response was signed and paired with an envelope containing: retrieved document IDs, embedding model version, redaction model version + threshold, generator prompt hash, tenant ID, and timestamp. The envelope went to an append-only audit log; the signature went to the client.
Why. Signing is the "explain this response three months from now" property. When a user asks why a particular sentence appeared, or when compliance runs a spot check, you need to reconstruct exactly which artifacts were in play. Rebuilding that from generic request logs is where audit stories quietly fall apart.
Observability, rollout, CI/CD
Three things around the pipeline mattered as much as anything inside it.
Structured logs, per-tenant latency breakdown. Every stage emitted a structured log line with stage, tenant, latency, and a query ID that stitched them together. The observability dashboard was per-tenant, not global, because "average latency is fine" hides the tenant whose filter selectivity is 40× worse than the median.
400+ user rollout. We didn't ship to 400 users on day one. It was three internal cohorts: an engineering cohort (loud, forgiving), a mixed cohort with real German-language docs (loud, less forgiving), then general availability. DKOM 2025 was the coming-out demo on top of the third cohort.
CI/CD deployment automation. Retrieval index rebuilds and redaction model checkpoint promotions were separately gated. The index rebuild pipeline could take hours; the redaction model was a fifteen-minute swap. Coupling them into one pipeline was the mistake we made once and unmade.
What broke first
The largest single production incident wasn't the LLM. It was the pre-filter.
Two months in, a tenant with strict retention rules started running queries that combined a tenant tag and a narrow date range (say, one week within a small tenant's slice of the corpus). Latency for that tenant jumped from the low hundreds of milliseconds into the multi-second range. p95 for the whole platform ticked up because their traffic wasn't small.
What happened is the failure mode I now call the burst filter. HNSW graph traversal picks candidate neighbors greedily; when a filter predicate is very selective, the traversal excludes most of the neighbors it encounters. If the filter is so selective that the candidate universe under it is smaller than k, the traversal has no valid stopping criterion inside its normal ef_search horizon. It keeps walking. In the pathological case it walked most of the reachable graph looking for k survivors that didn't exist, and the retrieval budget blew.
Two fixes, one short-term and one architectural.
Short-term: a pre-flight count on the filter predicate. If the candidate universe under the filter was below a threshold (empirically around 4× k), we fell back to an exact scan of that subset instead of using the graph at all. Exact scan of a small candidate set is trivially fast; graph traversal of a filtered-to-nothing candidate set is what breaks you.
Architectural: we started emitting the filter's selectivity into the audit envelope, so when a tenant hit the pathological regime again we saw it before they filed a ticket. The fix wasn't clever. It was that the graph index has a working regime, and the pre-filter can push you outside it, and you need to notice.
What I would do differently
Three things I would take into a v2.
Bake retrieval traces into the developer loop, alongside the audit log. The burst-filter incident would have shown up in a load test if the load test emitted per-stage traces the way production did. It didn't, because retrieval traces were a "compliance thing." That was the wrong bucket.
Separate the redaction model from the redaction policy. We shipped them together: one checkpoint, one entity taxonomy. Product teams over time wanted per-tenant policies (this tenant redacts internal employee names, this one doesn't). The policy should have been data, not code, from day one.
A first-class evaluation harness for the whole pipeline. We had per-component evals (retrieval recall, PII F1, generator quality), but the end-to-end eval was ad-hoc human review at each cohort promotion. A stable end-to-end eval set, versioned like code, would have caught two regressions we found in production first.
Closing
The lesson this system taught me, and the one I'll carry into the next: posture is a system property, not a component property. You cannot buy GDPR compliance from any single box in the diagram: not from the vector index, not from the PII detector, not from the LLM. What makes the system defensible is the shape of the boundaries: where the PII lives, which stage sees it, what leaves each stage in the audit envelope. Every one of the components in this post is replaceable; the shape isn't.
See also
- HNSW or IVF-PQ? What I Actually Chose at 2M Documents: the retrieval-index decision this architecture is built around.
- Redact at Retrieval, Not at Ingest: A GDPR-Compliant RAG Architecture: why the PII pass is between retrieval and generation, not upstream of ingest.
- DeBERTa over XLM-R for German PII: the choice of PII model and what German morphology cost us.