Module 3 · Retrieval that holds up · Chapter 20 · Free
Why basic RAG breaks, and which fix to try first
- A basic RAG (look up pages, then answer) assistant gets most questions right, and fails the rest for different reasons.
- There are six common ways it breaks. Each has a symptom you can see and a cheap first fix.
- Always look at which pages were found for a failing question before you change anything.
The one picture to remember: a pipeline from question to answer, with six failure spots pinned to it.
- 7 answers are good. He feels great.
- 3 answers are wrong, and each one sounds sure of itself.
- A teammate says, "Let's add a graph database and an agent." Another says, "Let's switch models."
Basic RAG (retrieval-augmented generation) works in 3 steps:
- Cut your documents into chunks. Give each chunk a list of numbers, a fingerprint of its meaning.
- Give the question a fingerprint too. Chunks with a similar fingerprint are close in meaning, so they are found.
- Hand the best few chunks to the AI model, which writes the answer from them.
Chapter 9, RAG end to end, teaches this in full.
- Learn the one habit that beats guessing: look at what was retrieved.
- Walk the map of six ways basic RAG breaks, and the cheapest fix for each.
- Do a sorting exercise: match a symptom to its likely cause.
- Follow Kavish as he works out the order of fixes, with real arithmetic.
- See two short scenarios, then check what you learned.
After this chapter you can:
- Look at a bad answer and say which of six causes it most likely has.
- Name the cheapest first fix for each cause, and why you try it before the fancy ones.
- Run a simple tally of failing questions to decide what to fix first.
Next: a few words you will meet all the way through. Each has a plain meaning and a small example.
Words you'll meet
- RAG
- Retrieval-augmented generation. Look up pages first, then answer from them. Example: find the leave policy, then answer "10 days".
- Chunk
- A small piece of a document that is stored and searched on its own. Example: one paragraph of the handbook.
- Rank
- The order of the search results, best first. Example: the right page sits at rank 1, or at rank 8.
- Hybrid search
- Search by meaning and by exact words together. Example: it finds "E-4410" by its letters and "money back" by its meaning.
- Re-ranker
- A slower second model that re-reads the top results and puts the best first. Example: it moves the page at rank 8 up to rank 1.
- Metadata filter
- A rule on facts stored with each chunk, such as version, date or who may read it. Example: "only the current version".
- Query rewriting
- Turning a vague or chatty question into a clear search. Example: "what about for them?" becomes "notice period for part-time staff".
- Multi-hop question
- A question whose answer needs several pages joined together. Example: "Which suppliers make parts used by Ohio customers?"
- Agentic loop
- The AI decides what to search, checks the results, and searches again if needed. Example: it tries once, sees a weak result, and rewrites the question.
- Knowledge graph
- Things joined by labelled links, stored so you can walk from one to another. Example: Customer, bought, Product, made by, Supplier.
Next: the habit that makes every later step easier.
Step 1: Look at what was retrieved, before you fix anything
A wrong answer has two parents. Either the AI was given the wrong pages, or it was given the right pages and misused them.
You cannot tell which from the answer alone. You can tell in about one minute.
The habit, for every failing question:
- Write down the question and the page that should answer it.
- Print the pages the system actually found, in order, with their rank (1 is best).
- Ask: was the right page there at all? If yes, at what rank? Was it complete?
Those three answers point at one of six causes. Most teams skip this and buy something fancy. A look at the retrieved pages is free.
Here is what that one-minute habit actually looks like. Kavish prints the results for a failing question about refunds:
Question: How many days do I have to return an item for a refund? Pages found, in order: 1. (score 0.81) Refund policy, 2022 version: "You have 30 days from delivery to return an item for a full refund." 2. (score 0.78) Returns FAQ: "Most items can be returned. See the refund policy for the time limits." 3. (score 0.74) Refund policy, 2025 version: "You have 14 days from delivery to return an item for a full refund." Answer the assistant gave: "You have 30 days to return an item for a full refund."
This is a realistic example printout, in the shape of a real one. Read it and the cause names itself: the current rule (14 days) was found, at rank 3, but the old rule won at rank 1 and the assistant trusted it. Nothing here is a model problem. It is card D, the wrong version, and it took one minute to see.
Think before, between and in the AI:
- Before the AI: the right page is missing. The search or the chunks failed (cards A, C).
- Between: the page is found but ranked low, the wrong version won, or the chunk lost its heading (cards B, C, D).
- In the AI: the page is on top and the answer is still wrong. Look at the instructions (card E).
- If no single page holds the answer, the design is the issue (card F).
One more thing. The AI can only be handed a few pages, so teams often send the top 3 to 5. This number is a setting you choose. This chapter uses made-up numbers such as top 3.
Next: the map. Six ways it breaks, each with a symptom and a cheap first fix.
Step 2: The map of six failures
Think of a hospital. Before treating anything, a doctor asks where it hurts. Here the body is a pipeline, and the map shows where each failure sits.
What you are looking at: the path of a question, top to bottom. Each failure card A to F is pinned to its stage. Press next to visit each card. Kavish's handbook assistant is the running example.
- You see: the right page is nowhere in the results. Kavish asks about error E-4410 and gets general guides.
Why: the question uses other words than the page, is vague, or the chunks were cut badly. The AI cannot use a page it never gets.
First fix: better chunks, then hybrid search (meaning plus exact words). - You see: the right page is at rank 6, but only the top 3 reach the AI. Kavish's part-time notice question does this.
Why: the first search is quick but rough. The top 3 is a setting you choose.
First fix: a re-ranker that re-reads the top results and sorts them better. - You see: the chunk says "You get 10 days." The heading that said who it applies to is in another chunk.
Why: a chunk is read alone, so it must carry its own context.
First fix: keep the heading or title with every chunk. - You see: an old policy outranks the new one, or a page shows up for someone who may not read it.
Why: search ranks by closeness and knows nothing about dates or permissions. That data must be stored with each chunk. A filter runs before ranking so a blocked page can never reach the AI.
First fix: metadata filters. - You see: the right page is at the top, but the answer is still wrong.
Why: the AI added its own guess, or got lost in a long prompt.
First fix: clearer instructions ("answer only from these pages, name them") and fewer, better pages. - You see: the answer is in no single page. "Which suppliers make parts used by Ohio customers?" is one.
Why: search returns pages one by one, not connections.
First fix: an agentic loop, or a graph. These cost the most, so they come last.
Where the picture breaks down:
- A doctor finds one cause. A RAG system often has two at once, such as A and D together.
- A real pipeline may order its stages a little differently. The map shows the idea, not your tool.
- The six cards are not equal in cost. A and B are often an afternoon of work. F can be weeks.
- The cards are a guide, not a law. Your tool may name or split them differently, so check its docs.
Next: try the map yourself. You will sort real-sounding symptoms into the part of the system that is most likely to blame.
Step 3: Try it: match each symptom to its cause
Why this exercise: in real work you meet the symptom first. Practice going from symptom to cause, so you reach for the right fix.
The rules you will use:
- If the right page was never retrieved, blame search or chunks.
- If it was retrieved but sits low, blame ranking.
- If it was retrieved but the wrong version, or the wrong person can see it, blame filters.
- If the right page was on top and the answer is still wrong, blame the model's instructions.
- If the page was found but its heading or context is missing, blame search or chunks too (card C).
- If no single page holds the answer, blame the design: it needs joined pages (card F).
These five buckets are broader than the six cards. Search and chunks cover A and C. The Ranking bucket gets only one item, and that is fine.
The limit: one symptom can have two causes. Pick the most likely, then check by looking at the retrieved pages.
Do this:
- Click a phrase to pick it up.
- Click the box that names the most likely cause.
- If you are wrong, read the hint, and try again.
- Do all seven. Notice how many hints ask where the page is.
- Two hints ask whether the page was found, and where it sits. You can only tell by looking at the retrieved pages.
- Only one item (the Ohio question) needs a bigger design. The other six are cheap to fix.
- Most teams reach for the expensive fix first, and miss the cheap ones.
Next: now see Kavish do it on his 3 failing questions, with a tally, to decide what to fix first.
Step 4: Kavish's tally, and which fix goes first
Kavish prints what was retrieved for each of his 3 failing questions. All numbers here are made up.
- Q4: "What does error E-4410 mean?" The right page is not in the top 10. Cause A: never retrieved.
- Q7: "Is there a notice period for part-timers?" The right page is at rank 6, and only the top 3 are used. Cause B: ranked low.
- Q9: "Which suppliers make parts used by Ohio customers?" No single page has the answer. Cause F: needs joined pages.
Now the arithmetic. These outcomes are made up, to show how to tally. Real fixes must be tested.
- Today: 7 good answers out of 10. That is 7 ÷ 10 = 70%.
- Say hybrid search fixes Q4 (an assumption). 8 ÷ 10 = 80%.
- Say a re-ranker fixes Q7 (also an assumption). 9 ÷ 10 = 90%.
- Q9 would still fail. 1 ÷ 10 = 10% of the questions need a bigger design.
So the answer is: fix A and B first. They lift the score from 70% to 90% with small, cheap changes. Q9 waits.
One pattern to carry into your own tally: in tallies like this, "never found" (A) and "wrong version" (D) together are usually more than half of all failures. Fix those two before buying anything.
Why wait on Q9? The fix is the costliest. It also helps only if many real users ask joined questions. Kavish checks the logs first. Only 1 question in 10 fails this way, so he wants more evidence first.
- Look at the retrieved pages.
- Check the instructions and prompt (E).
- Fix the data and chunks (A, C).
- Add hybrid search (A).
- Add a re-ranker (B).
- Rewrite vague queries (A). Chapter 21 covers this.
- Add metadata filters and permissions (D).
- Only then try an agentic loop or a graph (F).
The order follows cost. Cheap and common first. Big and rare last. The later chapters in the full course teach each rung in turn.
Predict: what should Kavish check?
Pick one answer, then see why.
The page was found and ranked first, so search and ranking are fine. A re-ranker or graph would fix a problem he does not have.
Next: two quick scenarios, one at a customer, to practise the habit.
Two scenarios
- A full-time employee asks about leave and gets "10 days", which is wrong for them.
- Printing the retrieved page shows the chunk has no heading.
- First fix: re-cut the handbook so each chunk keeps its heading. Cause C.
- Both fee pages match the question about equally well.
- Search cannot tell which is newer, because it ranks by closeness.
- First fix: store a "current version" flag with each page, and filter on it before ranking. Cause D.
Next: two quick checks on what you just learned.
Quick checks
Try it: debug this RAG
Why this exercise: so far, the symptoms were handed to you in words. Real debugging starts from a printout. Here are two failing questions, with the pages the system actually found. Name the card that is the root cause. A wrong pick earns a hint, not a mark. Keep going until the evidence convicts one card.
Case 1: the leave question
A new starter asks the handbook assistant about leave. The answer is out of date, and nobody changed the model.
Question: How many days of annual leave do new starters get? Pages found, in order: 1. (score 0.84) Staff handbook, 2023 version: "New starters receive 20 days of annual leave each year." 2. (score 0.80) Leave FAQ: "Your remaining leave balance is shown on the HR portal." 3. (score 0.77) Staff handbook, 2026 version: "New starters receive 25 days of annual leave each year." Answer the assistant gave: "New starters get 20 days of annual leave."
Case 2: the sick days question
A full-time employee asks about sick leave. The number in the answer is real. It just is not theirs.
Question: I work full-time. How many paid sick days do I get? Pages found, in order: 1. (score 0.84) Sick leave page, chunk 14: "You are entitled to 5 days of paid sick leave in each year of service." 2. (score 0.78) Sick leave page, chunk 15: "Sick leave must be reported to your manager before 9 a.m. on the first day." 3. (score 0.72) Sick leave page, chunk 2: "Part-time staff work three days a week or fewer." Answer the assistant gave: "You get 5 days of paid sick leave per year."
Kavish opens the original page to compare, the way Step 1 teaches. The "5 days" line sits under the heading "Part-time staff". Full-time staff get 10 days. No heading reached the assistant.
Notice what you did in both cases: you never touched the model. You read the pages, compared them with the source, and the cause named itself. That is the whole method.
Mix-ups to avoid
- "A wrong answer means a weak model." Often the model never saw the right page. Look at the retrieved pages.
- "More advanced means better." A graph or agent fixes only some failures, and costs far more. Cheap fixes come first.
- "One fix will cure everything." Each cause has its own fix. Fix one, run your test questions again, then decide.
Recap
- Look at what was retrieved before changing anything.
- Six ways it breaks: never found, ranked low, cut off, stages disagree, model misreads, needs many pages.
- Try cheap fixes first: chunks, hybrid search, re-ranker, query rewriting, filters. Agents and graphs come last.
- A later review of 30 failing questions shows many were old page versions and exact codes (numbers here are made up).
- A filter and hybrid search fix many of them in a week.
Keep the map itself, too. Copy the six cards onto one page and keep it beside your keyboard: A never found, B ranked low, C cut off, D wrong version or reader, E model misreads, F needs many pages. The next time an answer looks wrong, run down the checklist before you change anything.
Next: Chapter 21, "Better chunks and better questions". You now know where to look. The next chapter shows how to cut pages well and rewrite questions, the first and cheapest fixes on the list.
Review this chapter in 5 minutes
Watch the whole chapter recapped: the one-minute habit, the six failure cards, and the order of fixes.
Chapter quiz
Four questions. Score 70% or more to mark this chapter complete.