Module 2 · AI engineering foundations · Chapter 9 · Free
RAG end to end
- An AI knows the world but not your private files.
- RAG lets it look up the right pages just before it answers.
- Each stage of the lookup fixes a different problem.
The one picture to remember: an open-book exam with a Librarian (Search), a Guard (Permission check) and a Reader (Re-rank).
- In a customer demo it says, "You have 14 days to request a refund."
- The real policy says 30 days.
- It also quotes a contract line that only Legal may read.
- See why a fluent AI still gets your facts wrong, and the open-book fix.
- Follow one question through every stage, step by step.
- Try a lab where you switch the stages on and off.
- Do the scores by hand, so the lab is not magic.
- Learn which stage to blame when an answer is bad.
After this chapter you can:
- Explain in one minute what RAG is and why it beats retraining for company facts.
- Name the stages a question passes through, and what each one fixes.
- Say where a secret page must be blocked, and why a prompt rule is not enough.
Next: a few words you will see all the way through. Each has a plain meaning and a small example.
Words you'll meet
- Context limit (the desk)
- The most text the AI can read at once, so only a few pages fit (Chapter 7). Example: room for 3 pages, even if 10 matched.
- RAG
- Retrieval-augmented generation. Fetch the right pages first, then write the answer from them. Example: look up the refund page, then answer "30 days".
- Retrieval
- The "fetch pages" part of RAG. Example: searching 5,000 pages and returning the best 10.
- Shelving and index
- Shelving means cutting documents into chunks and filing them ahead of time. The index is the finding aid that makes search fast. Example: a book's back-of-book list.
- Access list
- A tag on each chunk saying who may read it. Example: "Legal" on a contract page.
- Chunk
- A page-sized piece of a document, about one topic. Example: the "Refunds" section of a long policy.
- Embedding
- Numbers that capture what a chunk means (Chapter 8). Example: "money back" and "refund" get nearby numbers.
- Keyword search
- Search that matches the exact letters you typed. Example: it finds "V-7731" only if the page holds "V-7731".
- Hybrid search
- Search by meaning and by exact words together, then mix the two scores. Example: 0.5 x meaning score + 0.5 x keyword score.
- Re-rank
- A careful second sort of the pages already found. Example: ten pages come back, and the re-ranker moves the best one to the top.
- Permission check
- Dropping pages the asker may not read, using an access list stored with each chunk. Example: a Legal-only page is removed for a support agent.
- Prompt
- The one message sent to the AI: rules, pages and question. Example: "Answer only from these pages. Page 1: ... Question: ..."
- Grounded
- An answer that sticks to the pages it was given, and names them. Example: "30 days (source: Refund policy v3)".
Next: first, why Kavish's assistant went wrong, and the simplest picture of the fix.
Step 1: Why a fluent AI still gets your facts wrong
Think of an exam. A student who has never worked at your company must answer a question about your refund policy.
- Asked cold, the student guesses. The guess sounds confident.
- Allowed to look through the right pages first, the student is usually correct.
That second way is RAG, an open-book exam.
Why not retrain the AI on your files instead?
- Looking things up is cheap and always current.
- Retraining is slow.
- Retrained knowledge goes stale the day a policy changes.
What you are looking at: four helpers and the shelves, in the exam. Press next to meet each one and see what it does.
On a phone, swipe the picture sideways to see all of it.
- The student has never seen your files. Asked cold, it would guess. So we do not let it answer yet.
- Your documents are filed once, in advance, as chunks. Each chunk is stamped with who may read it.
- The librarian takes the question to the shelves and brings back pages that look like a match.
- Pages the asker may not read are removed. This must happen before the AI sees anything.
- A slower, more careful reader puts the best page first. Only a few pages fit on the desk, the context limit.
- The student answers with the best pages open on the desk. This is the open-book exam.
Where the picture breaks down:
- A real librarian understands you. The search step only matches scores, so it can fetch a wrong page with total confidence.
- A real guard checks a badge at the door. In RAG the check must happen before the prompt is built. After that, the AI already holds the page.
- A student remembers the pages for later. The AI does not. Pages are looked up again for every question.
Next: the picture shows the idea. Now the same stages as a real system, with a question flowing through it.
Step 2: One question's journey through the system
The same helpers now have technical names. Follow one question from the top.
- Once, in advance: shelve the pages.
- Every question: search, check permission, re-rank, build the prompt, write the answer.
What you are looking at: boxes left to right, with the prompt and answer on the right. The dashed box is the setup. Press next to walk along. Each box keeps its name from Step 1.
On a phone, swipe the picture sideways. The prompt and answer boxes are on the far right.
- Split each document into chunks and turn them into embeddings (Chapter 8). Stamp each with who may read it. Why: you cannot search or filter what you never prepared.
- Someone logs in and types a question. The system already knows their role. Why it matters: the role drives step 3.
- Hybrid search: meaning catches paraphrases, exact keywords catch names and codes. Why both: each has a blind spot the other covers.
- Forbidden pages are dropped. An AI cannot repeat what it never received.
- A slower, more careful second reader sorts the pages. Why: only the top few fit in the prompt, so the order decides what the AI sees.
- Rules, top pages with sources, and the question go into one message. Remember Chapter 7: this all has to fit on the desk.
- It answers from the pages, names its sources, and says "I don't know" when they lack the answer.
Where this picture breaks down:
- Real systems may run extra steps, such as rewriting the question first. This is the core path.
- The arrows look instant. Each stage adds a little waiting time.
Next: reading is not the same as seeing it break. In the lab you switch stages on and off and watch the answer change.
Step 3: Try it: break it, then fix it
Why this exercise: to see for yourself what each stage fixes, and what only one stage can fix.
The lab holds four pages from a made-up company. The question is: "How long do Enterprise customers have to request a refund?"
Here are the four pages and their made-up scores (0 to 1, higher is better):
| Page | Meaning | Keyword | Careful reader |
|---|---|---|---|
| Refund policy v3 (current) | 0.74 | 0.95 | 0.97 |
| Refund policy v1 (old) | 0.82 | 0.55 | 0.30 |
| Marketing FAQ | 0.88 | 0.35 | 0.20 |
| Legal-only contract page | 0.79 | 0.60 | 0.55 |
The lab shows only the top 3 of these four. So a page can be hidden until a switch lifts it. The rules:
- Hybrid search off: pages are ranked by the meaning score alone. On: half meaning plus half keyword score.
- Re-ranker on: all four pages are re-sorted by the careful reader score.
- Permission filter on: pages the asker may not read are removed.
- After sorting, the top 3 of the four go to the AI. The first page decides the answer shown.
The limits: real scores come from models, and real stages are more complex. The lab shows the pattern, not exact numbers.
Do this (reload the page any time to start again):
- Leave all boxes off, asked by Support agent. Expect: a vague answer, and a red warning that a Legal-only page reached the prompt. The current policy is hidden, because it is not in the top 3.
- Tick Hybrid search. Expect: the current policy rises to first place and the answer says 30 days. The warning stays.
- Tick Permission filter, and leave it ticked from now on. Expect: the warning goes away.
- Untick Hybrid search. Expect: the vague answer returns.
- Tick Re-ranker. Expect: the current policy is first again, from a different stage.
- Switch Asked by to Legal. Expect: the Legal-only page may now appear, because Legal may read it.
- Only the permission filter keeps the Legal-only page out.
- Hybrid search rescues the current policy. So does the re-ranker, in this small lab, because it re-sorts every page it is given.
- In a real system the re-ranker only sees the pages search returned. If search never found the right page, the re-ranker cannot save it.
Next: the lab showed a rule, "half meaning, half keywords". Let us do that sum by hand, so you can trust it.
Step 4: Do the scores by hand
Meaning alone picked the marketing FAQ. Adding exact keywords pulled up the current policy, so the answer became 30 days.
Let's work it out. The scores are the lab's made-up numbers. Each is between 0 and 1.
- Question: "How long do Enterprise customers have to request a refund?"
- Meaning only: FAQ 0.88, old policy 0.82, current policy 0.74. The FAQ wins. It names no days.
- Now mix half meaning and half keywords.
- Current policy: 0.5 × 0.74 + 0.5 × 0.95 = 0.37 + 0.475 = 0.845.
- Old policy: 0.5 × 0.82 + 0.5 × 0.55 = 0.41 + 0.275 = 0.685.
- Marketing FAQ: 0.5 × 0.88 + 0.5 × 0.35 = 0.44 + 0.175 = 0.615.
- The careful reader gives its own score: current policy 0.97, old policy 0.30, FAQ 0.20. Re-rank puts the current policy first.
So the answer is 30 days, taken from the current policy.
Check the old policy row yourself
Old policy: 0.5 × 0.82 + 0.5 × 0.55 = 0.41 + 0.275 = 0.685. It beats the FAQ (0.615) but loses to the current policy (0.845).
Optional depth: why page size matters
- Chunks that are too small lose context. "30 days" alone does not say which plan.
- Chunks that are too big bury the answer in noise.
- Try a few sizes and test them on real questions.
Next: you can now fix a demo. In real work you get a bad answer and must find the cause. A second example, then a map of which stage to check.
Step 5: When the answer is bad, which stage do you blame?
First, a second case where meaning search alone fails.
- The code is an exact string, so meaning search is weak here.
- Keyword search finds the page that holds "V-7731".
- Hybrid search keeps both strengths.
Now the map. Each stage has its own typical failure. Check them in order. Why this order: each stage feeds the next, so an early fault hides later ones.
On a phone, swipe the picture sideways to see all five stages.
What you are looking at: the five stages on top, and the symptom each one causes underneath. Press next to go stage by stage.
- Chunks too small lose context. Chunks too big bury the answer. Fix: test a few chunk sizes on real questions.
- Meaning search is weak on codes like "V-7731". Fix: add keyword search, so you have hybrid search.
- Nothing earlier knows who is asking. Fix: store an access list with every chunk and filter before the prompt is built.
- Only the top few pages fit in the prompt. Fix: add a re-ranker so the best page lands in the top few.
- Retrieval was fine, but the AI went beyond its pages. Fix: tell it to answer only from the pages, name sources, and say "I don't know".
Where the picture breaks down:
- Symptoms overlap. A wrong number could come from a stale page (search) or from the AI inventing it (writing). Read the pages in the prompt first.
- Stages affect each other. A bad chunk size makes search and re-rank look worse than they are.
Next: one prediction and two quick quizzes on what you just learned, then common mix-ups and a final four-question quiz.
A page the AI never receives cannot be repeated.
Quick checks
Mix-ups to avoid
- "RAG teaches the AI our documents." No. Pages are looked up for each question, then forgotten.
- "A rule in the prompt keeps secrets safe." No. Telling the AI "never reveal secrets" is a request, not a lock.
- "A fluent answer is a correct answer." No. Check the pages behind it and the sources it names.
Recap
- RAG is an open-book exam: look up the right pages first, then answer from them with sources.
- Hybrid search fixes missed exact words. Re-rank fixes order. Only the permission filter prevents leaks.
- For a bad answer, check the stages one by one: shelving, search, permission check, re-rank, grounding.
- The intern asks about pay bands.
- The assistant answers from an executive memo. The answer is correct.
- The memo was never meant for the intern.
Next: Chapter 10, "Prompt engineering that survives production". You now know how the right pages reach the AI. Chapter 10 covers how to write the instructions around them, so the answer stays grounded when real users arrive.
Chapter quiz
Four questions. Score 70% or more to mark this chapter complete.