Module 3 · Retrieval that holds up · Chapter 24 · Free
Agentic RAG: letting the model decide when and where to search
- Basic RAG searches once, then answers, even when the search was poor.
- Agentic RAG lets the model choose a source, check the pages, and try again.
- Every extra try costs time and money, so you cap the tries and log each one.
The one picture to remember: a research assistant who checks the first shelf, says "this is not it", and walks to a different shelf. Then gives up politely after three tries.
- A manager asks: "Does the new parental leave apply to contractors, and what changed since last year?"
- The assistant searches once and finds three pages about leave. None mention contractors.
- It writes a calm, confident answer anyway. The answer is wrong.
Basic RAG (retrieval-augmented generation) is a lookup before an answer. Your documents are cut into chunks and turned into embeddings, which are numbers for meaning. A question is searched by meaning, and the best chunks go to the model. The model writes the answer from them. Chapter 9, RAG end to end, teaches this in full. You only need the idea above to read on.
- See what "agentic" means, and why one search is not always enough.
- Follow the loop step by step in a clickable picture.
- Play Kavish in a role-play and make the choices yourself.
- Work out what the extra loops cost, with worked arithmetic.
- Learn how the loop breaks, and when not to use it at all.
After this chapter you can:
- Explain in one minute how agentic RAG differs from basic RAG.
- Draw the loop: decide, pick a source, search, grade, rewrite, answer or give up.
- Say what a step cap and a log are for, and when a plain single search is the better choice.
Next: a few words you will see all the way through. Each has a plain meaning and a small example.
Words you'll meet
Skim this list for now. Every word here is explained again where it is used.
- Agentic
- The model makes some of the choices, instead of following one fixed path. Example: it decides to search a second time.
- Agent loop
- A cycle the model repeats: act, look at the result, decide what to do next. Example: search, read, search again.
- Source
- One place the assistant can look things up. Example: the HR handbook, the ticket history, or a database of orders.
- Routing
- Choosing which source or tool fits the question. Example: a pay question goes to the HR handbook, not the ticket history.
- Grader
- A check, often another model call, that asks "do these pages answer the question?" Example: it says "no, nothing about contractors".
- Query rewriting
- Turning a question into a better search. Example: "Do contractors get it?" becomes "contractor eligibility for parental leave".
- Sub-question
- One small piece of a bigger question. Example: "What changed in the leave policy?" is half of the manager's question.
- Step cap
- A hard limit on how many loops are allowed. Example: at most 3 searches, then stop.
- Log (trace)
- A saved record of every step the agent took. Example: "search 2: handbook, query X, graded not enough".
- Tool
- Something the model can call to do a job. Example: a search tool that looks in the handbook.
- Model call
- One request sent to the AI model. Example: one grading check is one model call.
- Prompt
- The written instructions you give the model. Example: "Answer only from these pages."
- Chunk
- A small piece of a document, cut for search. Example: one page of the handbook.
- Sample
- A small set of real cases you check by hand. Example: 50 real questions.
- Citation
- A named source for a claim, so a reader can check it. Example: "Contractor handbook, page 4".
Related ideas you may hear: corrective RAG and self-checking RAG. They focus on the grade-and-retry part.
Next: first, what changes when the model gets to choose, and the picture that shows it.
Step 1: Give the assistant a loop, not a single shot
Think of a research assistant. Basic RAG is an assistant who checks one shelf and answers. Agentic RAG is one who can look, think, and look again.
The assistant gets these new powers:
- Decide whether to search at all. A "hello" needs no lookup.
- Pick a source. Handbook, tickets, database, or the web.
- Grade what came back. Is it enough?
- Rewrite and retry when it is not enough.
- Stop with a sourced answer, or with "I could not find it".
What you are looking at: the loop for the manager's question. The top row is the first try. The bottom row is what happens when the pages are not good enough. Press next to walk along.
On a phone, swipe the picture sideways to see all of it. Grade and Answer sit on the right. Tap Next and swipe to follow each step.
- "Does the new parental leave apply to contractors, and what changed since last year?" It has two parts, so one search may not cover both.
- The model asks itself if it needs a lookup. A greeting does not. A company policy does. Why: skipping needless searches saves time and money.
- This is routing. Staff leave rules live in the handbook. Contractor terms live in a separate contractor guide. The model picks one to try first.
- The chosen source is searched, the same way as basic RAG. Why: the loop is built on top of retrieval, not instead of it.
- The grader reads the question and the pages. It asks, "Is the answer here?" Why: this is the step a single-shot system skips.
- The pages say nothing about contractors. The model rewrites the search ("contractor eligibility for parental leave"). Then it picks a source again, here the contractor guide. A retry may keep the same source, but a new one often helps. Each trip around the loop counts toward the cap.
- Good pages found. The model answers from them and names each page it used, so a person can check.
- The cap is 3 tries, a common starting point. Tune it on your own questions. When the tries are used up, the model stops. "I could not find this. Here is what I did find." Why: an honest gap beats a confident guess.
Where the picture breaks down:
- A real assistant knows when it is confused. A model only guesses from the pages, and the grader can be wrong.
- A person tires and stops. A loop does not tire, so you must set the stop yourself.
- The arrows look instant. Every box is a model call or a search, and each one takes time. Step 3 shows the sums.
The grader is not magic. It is one more model call, with a short, strict prompt. Here is a realistic one, in the shape teams actually use:
You are checking whether some retrieved pages answer a question.
Question: {the question}
Pages:
{the pages the search found}
Reply with one verdict only:
ENOUGH — the pages contain everything needed to answer the question.
NOT ENOUGH — something is missing. Name what is missing in one short line.
Do not answer the question yourself. Judge only from the pages above.
Notice what the prompt forbids: answering the question. A grader that answers instead of judging hides the very gap it exists to catch.
Next: you have seen the loop. Now make the choices yourself, as Kavish, and watch the quality and the cost move.
Step 2: Try it: you are Kavish, with the loop in your hands
Why this exercise: to feel the trade-off. More loops can raise answer quality, and they always raise cost and wait.
The rules of this role-play:
- You pick a reply at each turn. The story is made up.
- "Answer quality" goes up when you check the pages, route well, and cite sources.
- "Cost and wait" goes up with every extra search or grading call. Lower is better.
- The meters are a teaching score only. The author set the numbers. They are not computed or measured.
- You click one reply button each turn. The whole play takes about 2 minutes and has 5 endings.
Do this (press "Try again" at the end to replay):
- Play once the careless way: one search, then ship it. Note the meters.
- Play again. This time add a grader, and rewrite when it says "not enough".
- Look at the turn after the grader says "not enough". Find the choice that spends more and gets nothing back.
- Find the ending that says "I could not find it". It comes after you repeat a search. Decide if that is good or bad. "What to notice" answers it.
- The best ending costs more than the careless one. Quality is bought with extra calls.
- Retrying the same search was wasted money. A retry should change the question or the source.
- "I could not find it" can be a success, not a failure.
Next: the role-play used cost as a meter. Let us do the worked arithmetic with made-up numbers, so you can talk about cost with a customer.
Step 3: Count what the extra loops cost
Every box in the loop is a search or a model call. So every loop adds wait and money.
To keep the sums simple, we count only search, grade and answer. We ignore the decide, route and rewrite calls. They add a little more.
These numbers are made up. Check your own tools for real ones.
- One search takes 0.3 seconds and costs 0.0005 dollars.
- One grading call takes 1.0 second and costs 0.001 dollars.
- The final answer call takes 1.5 seconds and costs 0.002 dollars.
Basic RAG (one search, no grading, one answer):
- Time: 0.3 + 1.5 = 1.8 seconds.
- Cost: 0.0005 + 0.002 = 0.0025 dollars.
Agentic, 1 loop (search and grade once, then answer):
- One loop: 0.3 + 1.0 = 1.3 seconds, and 0.0005 + 0.001 = 0.0015 dollars.
- Time: 1.3 + 1.5 = 2.8 seconds.
- Cost: 0.0015 + 0.002 = 0.0035 dollars.
Agentic, 3 loops (the cap is reached):
- Three loops: 3 × 1.3 = 3.9 seconds, and 3 × 0.0015 = 0.0045 dollars.
- Time: 3.9 + 1.5 = 5.4 seconds.
- Cost: 0.0045 + 0.002 = 0.0065 dollars.
Now compare. Time: 5.4 ÷ 1.8 = 3 times slower. Cost: 0.0065 ÷ 0.0025 = 2.6 times costlier.
Scale it up. At 10,000 questions a day (an example volume, made up):
- Basic: 10,000 × 0.0025 = 25 dollars a day.
- Three loops every time: 10,000 × 0.0065 = 65 dollars a day.
So the answer is: in this made-up case, a full three-loop run waits 5.4 seconds and costs 2.6 times more. Many questions may need fewer loops, so the real average may sit in between. Measure yours.
Next: cost is the visible risk. The loop has quieter ways to fail, and some questions should never use it.
Step 4: How the loop breaks, and when to skip it
Four ways it fails:
- It never ends. The model keeps trying. Fix: a hard step cap, for example 3.
- The grader is too trusting. It says "good enough" for weak pages. Fix: check the grader against human judgement on a sample, for example 50 real questions you check by hand.
- The agent reads what the asker may not. A search tool can reach a lot. Fix: the agent may only search what the asker may read. Enforce it in the search tool, not in the prompt (the written instructions).
- You cannot see what happened. Fix: log every step: the source, the query, the pages and the grade.
Here is what one logged run looks like, for the manager's question from the start of this chapter. The cap is 3 tries. This run needed 2:
question: "Does the new parental leave apply to contractors, and what changed since last year?" loop 1 of 3 — source: staff handbook — query: "parental leave policy changes" pages found: "Parental leave" (page 12), "Leave types" (page 9), "Returning to work" (page 15) grade: NOT ENOUGH — the pages say what changed (12 to 16 weeks) but say nothing about contractors loop 2 of 3 — rewrite: "contractor eligibility for parental leave" — source: contractor guide pages found: "Contractor handbook" (page 4): contractors are not covered by the staff leave policy grade: ENOUGH — both parts of the question are now covered answer: "Parental leave for staff rose from 12 to 16 weeks. Contractors are not covered by the staff policy; a different process applies." (Sources: staff handbook, page 12; contractor handbook, page 4)
With a log like this, a bad answer stops being a mystery. You can point at the exact loop where it went wrong.
When not to use it:
- Simple lookups. "What is the office wifi password policy?" One good search is enough.
- Strict wait limits. A live voice call often cannot wait that long.
- When basic RAG is already failing. Fix the chunks and the search first. A loop on a weak search just fails more slowly.
A useful middle path: run one search, grade it, and only loop if the grade says "not enough". Easy questions then stay fast.
- Half the questions are "when does the Leeds shop open?" Those need one lookup.
- One in ten is a warranty puzzle across a manual, an order record and a policy.
- Kavish routes the simple ones to a single search, and lets only the puzzles enter the loop, with a cap of 3.
Next: two quick checks and a prediction, then common mix-ups, and the final four-question quiz.
The cap is a dial. Low is cheap and fast, but hard questions lose their second chance. Pick it by testing on real questions.
Mix-ups to avoid
- "Agentic RAG is always better." No. For simple lookups it is slower and costlier, with no gain.
- "The grader is always right." No. It is a model too. Check it against human judgement.
- "More loops means a better answer." No. If the source holds no answer, more tries only burn money. Know when to stop.
- "The agent can search anything, so security is solved." No. The agent should only reach what the asker may read.
Recap
- Agentic RAG adds choices to retrieval: decide, route, grade, rewrite, then answer or give up.
- Each loop adds model calls, wait and money. Cap the steps and log every one.
- Skip it for simple lookups and tight time limits. Fix basic search before you add a loop.
- It answers well in demos.
- In real use, some questions run for twenty steps and the bill spikes.
- Nobody can say why, because nothing was logged.
One search, no loop: "Yes, the new parental leave applies to contractors, and it rose from 12 to 16 weeks." Confident, calm, and wrong. The pages behind it never mentioned contractors.
With the loop: "Parental leave for staff rose from 12 to 16 weeks (staff handbook, 'Parental leave'). Contractors are not covered by the staff policy; a different process applies (contractor handbook, page 4)." Slower, and a reader can check every claim.
One limit to keep in mind. The loop can judge whether it has enough pages. It cannot judge whether its final answer is actually right: a smooth answer can still misread good pages. Checking the answers themselves is an evaluation problem, and later chapters in this course teach it.
Next: the next chapter, "Knowledge graphs and GraphRAG: when relationships matter". A loop helps when one search is not enough. Some questions ask how things connect, and a graph is built for that. The rest of the module also shows how to choose among all these tools, and how to measure whether any of them helped.
Review this chapter in 5 minutes
Watch the whole chapter recapped: the loop, the grader, what the extra tries cost, and when to skip it.
Chapter quiz
Four questions. Score 70% or more to mark this chapter complete.