What RAG Is (and the Problem It Solves)
- Say what the three letters R-A-G stand for, and what each word means
- Understand the exact problem RAG was built to solve
- Watch retrieval happen — and see why a good system sometimes says "I don't know"
The problem, first
A large language model is trained once, on data with a cutoff date. That gives it a problem with two faces:
- It doesn't know your stuff. Your company handbook, your product docs, your database, last week's meeting notes — none of that was in its training data.
- When it doesn't know, it often guesses anyway. A confident, wrong answer — a hallucination — is the most damaging thing an AI product can do.
You could retrain the model on your data, but that's slow, expensive, and out of date the moment a document changes. There's a far simpler idea.
The idea: look it up, then answer
RAG stands for Retrieval-Augmented Generation. Read it right to left and it explains itself:
- Generation — the model writes the answer (what LLMs do).
- Augmented — but we augment it with extra information first.
- Retrieval — that information is retrieved from your own documents, chosen because it's relevant to the question.
That open-book framing is the whole thing. Instead of asking the model to recall a fact it may never have known, you put the source on the desk in front of it and ask it to answer from that. It can now cite where the answer came from — and admit when the answer isn't in the book.
See it happen
Enough words. Below is a tiny "company handbook." Ask it a question and watch which passages get retrieved — the ones the model would then answer from. Try the suggestions, then try your own. (Notice it matches meaning: ask about "vacation" and it finds "leave.")
Employees accrue 25 days of paid leave per year, accrued monthly.
Remote work is allowed up to 3 days per week with manager approval.
Parental leave is 16 weeks, fully paid, for every parent.
Sick leave requires no doctor's note for the first 3 days.
The meal reimbursement limit when travelling is 40 EUR per day.
The Eindhoven office is open 08:00-19:00 on weekdays.
Laptops are replaced every 3 years; request a swap via the IT portal.
The 3 green passages are sent to the model as context. It answers only from these — and cites them. Notice it matches meaning: “vacation” finds “leave” even though the words differ.
Illustrative: a tiny concept map stands in for real embeddings.
Try the last suggestion — "What is the dental insurance plan?" — and watch what happens: nothing relevant comes back, and a good RAG system answers “I don't know” rather than inventing a plan. That honest refusal is exactly the behaviour that makes RAG trustworthy, and we'll keep coming back to it.
- LLMs don't know your private/changing data, and tend to guess when they don't know
- RAG fixes this by retrieving relevant passages and having the model answer from them — an open-book exam
- Retrieval matches meaning, not just keywords, and enables citations
- When nothing relevant is retrieved, the right answer is an honest "I don't know"
- In your own words, what do the three words in "Retrieval-Augmented Generation" each contribute?
- Why is RAG a better fit than retraining the model when your documents change every week?