Deployed Lab

Module 2 · AI engineering foundations · Chapter 9 · Free

RAG end to end

12 min3 clickable pictures1 retrieval lab7 checks
In 30 seconds

The one picture to remember: an open-book exam with a Librarian (Search), a Guard (Permission check) and a Reader (Re-rank).

The problem Kavish builds a support assistant (a composite example). Its English is fluent. It was simply handed the wrong pages.
The plan for this chapter
  1. See why a fluent AI still gets your facts wrong, and the open-book fix.
  2. Follow one question through every stage, step by step.
  3. Try a lab where you switch the stages on and off.
  4. Do the scores by hand, so the lab is not magic.
  5. Learn which stage to blame when an answer is bad.

After this chapter you can:

Next: a few words you will see all the way through. Each has a plain meaning and a small example.

Words you'll meet

Context limit (the desk)
The most text the AI can read at once, so only a few pages fit (Chapter 7). Example: room for 3 pages, even if 10 matched.
RAG
Retrieval-augmented generation. Fetch the right pages first, then write the answer from them. Example: look up the refund page, then answer "30 days".
Retrieval
The "fetch pages" part of RAG. Example: searching 5,000 pages and returning the best 10.
Shelving and index
Shelving means cutting documents into chunks and filing them ahead of time. The index is the finding aid that makes search fast. Example: a book's back-of-book list.
Access list
A tag on each chunk saying who may read it. Example: "Legal" on a contract page.
Chunk
A page-sized piece of a document, about one topic. Example: the "Refunds" section of a long policy.
Embedding
Numbers that capture what a chunk means (Chapter 8). Example: "money back" and "refund" get nearby numbers.
Keyword search
Search that matches the exact letters you typed. Example: it finds "V-7731" only if the page holds "V-7731".
Hybrid search
Search by meaning and by exact words together, then mix the two scores. Example: 0.5 x meaning score + 0.5 x keyword score.
Re-rank
A careful second sort of the pages already found. Example: ten pages come back, and the re-ranker moves the best one to the top.
Permission check
Dropping pages the asker may not read, using an access list stored with each chunk. Example: a Legal-only page is removed for a support agent.
Prompt
The one message sent to the AI: rules, pages and question. Example: "Answer only from these pages. Page 1: ... Question: ..."
Grounded
An answer that sticks to the pages it was given, and names them. Example: "30 days (source: Refund policy v3)".

Next: first, why Kavish's assistant went wrong, and the simplest picture of the fix.

Step 1: Why a fluent AI still gets your facts wrong

Think of an exam. A student who has never worked at your company must answer a question about your refund policy.

That second way is RAG, an open-book exam.

Why not retrain the AI on your files instead?

What you are looking at: four helpers and the shelves, in the exam. Press next to meet each one and see what it does.

On a phone, swipe the picture sideways to see all of it.

RAG as an open-book exam The student asks a question. The librarian fetches pages from the shelves, which hold your documents. A door guard blocks pages the student may not read. A careful reader puts the best page first and passes the pages back to the student, who writes the answer. The studentthe AI, asks The librarian= Search The door guard= Permission check A careful reader= Re-rank The shelvesyour documents best pages go back to the student
  1. The student has never seen your files. Asked cold, it would guess. So we do not let it answer yet.
  2. Your documents are filed once, in advance, as chunks. Each chunk is stamped with who may read it.
  3. The librarian takes the question to the shelves and brings back pages that look like a match.
  4. Pages the asker may not read are removed. This must happen before the AI sees anything.
  5. A slower, more careful reader puts the best page first. Only a few pages fit on the desk, the context limit.
  6. The student answers with the best pages open on the desk. This is the open-book exam.
The AISearchPermission checkRe-rankYour data

Where the picture breaks down:

Next: the picture shows the idea. Now the same stages as a real system, with a question flowing through it.

Step 2: One question's journey through the system

The same helpers now have technical names. Follow one question from the top.

What you are looking at: boxes left to right, with the prompt and answer on the right. The dashed box is the setup. Press next to walk along. Each box keeps its name from Step 1.

On a phone, swipe the picture sideways. The prompt and answer boxes are on the far right.

The RAG question path A question goes to a search step that fetches pages by keywords and meaning, then a badge check removes pages the person may not read, then a re-order step puts the best page first. The rules, pages and question are combined into one prompt, and the AI writes an answer with sources. A setup step, done once, shelves the pages. Setup, once: shelvesplit, index, tag who may read Question Searchthe librarian Permissioncheckthe guard Re-rankthe reader Build the promptrules, pages, question AI writes answerwith sources
  1. Split each document into chunks and turn them into embeddings (Chapter 8). Stamp each with who may read it. Why: you cannot search or filter what you never prepared.
  2. Someone logs in and types a question. The system already knows their role. Why it matters: the role drives step 3.
  3. Hybrid search: meaning catches paraphrases, exact keywords catch names and codes. Why both: each has a blind spot the other covers.
  4. Forbidden pages are dropped. An AI cannot repeat what it never received.
  5. A slower, more careful second reader sorts the pages. Why: only the top few fit in the prompt, so the order decides what the AI sees.
  6. Rules, top pages with sources, and the question go into one message. Remember Chapter 7: this all has to fit on the desk.
  7. It answers from the pages, names its sources, and says "I don't know" when they lack the answer.
UserSearchPermission checkModel stagesYour data and prompt

Where this picture breaks down:

Next: reading is not the same as seeing it break. In the lab you switch stages on and off and watch the answer change.

Step 3: Try it: break it, then fix it

Why this exercise: to see for yourself what each stage fixes, and what only one stage can fix.

The lab holds four pages from a made-up company. The question is: "How long do Enterprise customers have to request a refund?"

Here are the four pages and their made-up scores (0 to 1, higher is better):

PageMeaningKeywordCareful reader
Refund policy v3 (current)0.740.950.97
Refund policy v1 (old)0.820.550.30
Marketing FAQ0.880.350.20
Legal-only contract page0.790.600.55

The lab shows only the top 3 of these four. So a page can be hidden until a switch lifts it. The rules:

The limits: real scores come from models, and real stages are more complex. The lab shows the pattern, not exact numbers.

Do this (reload the page any time to start again):

  1. Leave all boxes off, asked by Support agent. Expect: a vague answer, and a red warning that a Legal-only page reached the prompt. The current policy is hidden, because it is not in the top 3.
  2. Tick Hybrid search. Expect: the current policy rises to first place and the answer says 30 days. The warning stays.
  3. Tick Permission filter, and leave it ticked from now on. Expect: the warning goes away.
  4. Untick Hybrid search. Expect: the vague answer returns.
  5. Tick Re-ranker. Expect: the current policy is first again, from a different stage.
  6. Switch Asked by to Legal. Expect: the Legal-only page may now appear, because Legal may read it.
What to notice

Next: the lab showed a rule, "half meaning, half keywords". Let us do that sum by hand, so you can trust it.

Step 4: Do the scores by hand

Meaning alone picked the marketing FAQ. Adding exact keywords pulled up the current policy, so the answer became 30 days.

Let's work it out. The scores are the lab's made-up numbers. Each is between 0 and 1.

  1. Question: "How long do Enterprise customers have to request a refund?"
  2. Meaning only: FAQ 0.88, old policy 0.82, current policy 0.74. The FAQ wins. It names no days.
  3. Now mix half meaning and half keywords.
  4. Current policy: 0.5 × 0.74 + 0.5 × 0.95 = 0.37 + 0.475 = 0.845.
  5. Old policy: 0.5 × 0.82 + 0.5 × 0.55 = 0.41 + 0.275 = 0.685.
  6. Marketing FAQ: 0.5 × 0.88 + 0.5 × 0.35 = 0.44 + 0.175 = 0.615.
  7. The careful reader gives its own score: current policy 0.97, old policy 0.30, FAQ 0.20. Re-rank puts the current policy first.

So the answer is 30 days, taken from the current policy.

Check the old policy row yourself

Old policy: 0.5 × 0.82 + 0.5 × 0.55 = 0.41 + 0.275 = 0.685. It beats the FAQ (0.615) but loses to the current policy (0.845).

Optional depth: why page size matters

Next: you can now fix a demo. In real work you get a bad answer and must find the cause. A second example, then a map of which stage to check.

Step 5: When the answer is bad, which stage do you blame?

First, a second case where meaning search alone fails.

Scenario 2: the part number A repair shop (a made-up example) searches for "valve V-7731". Meaning search returns general valve guides.

Now the map. Each stage has its own typical failure. Check them in order. Why this order: each stage feeds the next, so an early fault hides later ones.

On a phone, swipe the picture sideways to see all five stages.

What you are looking at: the five stages on top, and the symptom each one causes underneath. Press next to go stage by stage.

Which stage to check for each symptom Five stages in order: shelving, search, permission check, re-rank, writing. Under each is its typical symptom. Shelving: the right page is never found or lacks context. Search: an exact code or name is not found. Permission check: the asker sees a page they must not read. Re-rank: the right page is found but ranked too low to fit. Writing: a number appears that is in no page. Shelving Search Permissioncheck Re-rank Writing Right page neverfound, or lacksits context Code or namenot found Asker sees apage they mustnot read Right page foundbut ranked toolow to fit A number appearsthat is in nopage
  1. Chunks too small lose context. Chunks too big bury the answer. Fix: test a few chunk sizes on real questions.
  2. Meaning search is weak on codes like "V-7731". Fix: add keyword search, so you have hybrid search.
  3. Nothing earlier knows who is asking. Fix: store an access list with every chunk and filter before the prompt is built.
  4. Only the top few pages fit in the prompt. Fix: add a re-ranker so the best page lands in the top few.
  5. Retrieval was fine, but the AI went beyond its pages. Fix: tell it to answer only from the pages, name sources, and say "I don't know".
Symptom

Where the picture breaks down:

Next: one prediction and two quick quizzes on what you just learned, then common mix-ups and a final four-question quiz.

A page the AI never receives cannot be repeated.

Quick checks

Mix-ups to avoid

Recap

In the field A consultant (a composite example) demos a company assistant to a new intern. The costliest RAG bug is a correct answer built from a document the asker was never meant to see. Store the access list with every chunk from day one, and test while logged in as a low-privilege user.

Next: Chapter 10, "Prompt engineering that survives production". You now know how the right pages reach the AI. Chapter 10 covers how to write the instructions around them, so the answer stays grounded when real users arrive.

Chapter quiz

Four questions. Score 70% or more to mark this chapter complete.

Review