Pranav Srivastava

10 lessons

0/10 done
Lesson 9 of 10·16 min·Advanced

The Strange Scenarios (Where Naïve RAG Breaks)

What you will learn
  • Handle contradictory and stale sources without silently picking wrong
  • Enforce access control at retrieval time — the security case people forget
  • Refuse out-of-scope questions and cope with empty or huge retrieval
  • Deal with duplicates, tables, and adversarial documents deliberately

This is where naïve RAG quietly breaks. Every one of these will happen in production — design for them now, not after the incident.

Stale & contradictory sources

Two documents disagree — an old policy and a new one. Filter by recency, prefer authoritative sources, and when a genuine conflict remains, have the model surface both with their dates rather than silently pick one.

Access control & multi-tenancy

PII

Detect and redact sensitive data at ingestion, and again on output. Know what you're storing and where — and be able to delete a person's data on request.

Out-of-scope, empty, and huge retrieval

  • Out-of-scope questions. Users ask things your corpus can't answer. A confidence threshold on retrieval scores — plus the honest "I don't know" from Module 7 — keeps the system from bluffing. (You watched this in the Module 1 playground.)
  • Empty retrieval. No good matches → say so; don't force an answer from weak context.
  • Huge retrieval. Hundreds of matches → re-rank hard and cap k; never dump everything into the window.

Duplicates, tables, and adversaries

  • Near-duplicate chunks. The same boilerplate appears in fifty documents and floods your top-k. De-duplicate at ingestion and diversify results (e.g. MMR) so the context isn't five copies of one sentence.
  • Tables & numbers. Prose-tuned embeddings are weak on tabular data. Keep tables intact, handle them deliberately, and never trust a number the model "remembered" — cite the row.
  • Adversarial documents. Content designed to manipulate the model (the injection from Module 7). Sanitise inputs, sandbox permissions, and validate outputs.
Chapter summary
  • Resolve contradictions by recency and authority; surface genuine conflicts with dates
  • Enforce access control at query time — a security boundary, build it first
  • Redact PII in and out; threshold retrieval to refuse out-of-scope questions
  • De-duplicate and diversify; handle tables and adversarial documents deliberately
Check your understanding
  1. Your RAG serves multiple companies from one index. What must happen before ranking, and why is it non-negotiable?
  2. Two documents give different answers because one is outdated. What's a better response than silently returning one of them?

Finished this lesson?

Mark it done — your progress is saved automatically.