Deployed Lab

Module 3 · Retrieval that holds up · Chapter 24 · Free

Agentic RAG: letting the model decide when and where to search

12 min1 clickable picture1 role-play7 checks5-min video review
In 30 seconds

The one picture to remember: a research assistant who checks the first shelf, says "this is not it", and walks to a different shelf. Then gives up politely after three tries.

The problem Kavish builds a staff help assistant for a made-up company (a composite example). One search was never going to be enough. A person would have looked in a second place. Can the assistant learn to do that?
Quick refresher

Basic RAG (retrieval-augmented generation) is a lookup before an answer. Your documents are cut into chunks and turned into embeddings, which are numbers for meaning. A question is searched by meaning, and the best chunks go to the model. The model writes the answer from them. Chapter 9, RAG end to end, teaches this in full. You only need the idea above to read on.

The plan for this chapter
  1. See what "agentic" means, and why one search is not always enough.
  2. Follow the loop step by step in a clickable picture.
  3. Play Kavish in a role-play and make the choices yourself.
  4. Work out what the extra loops cost, with worked arithmetic.
  5. Learn how the loop breaks, and when not to use it at all.

After this chapter you can:

Next: a few words you will see all the way through. Each has a plain meaning and a small example.

Words you'll meet

Skim this list for now. Every word here is explained again where it is used.

Agentic
The model makes some of the choices, instead of following one fixed path. Example: it decides to search a second time.
Agent loop
A cycle the model repeats: act, look at the result, decide what to do next. Example: search, read, search again.
Source
One place the assistant can look things up. Example: the HR handbook, the ticket history, or a database of orders.
Routing
Choosing which source or tool fits the question. Example: a pay question goes to the HR handbook, not the ticket history.
Grader
A check, often another model call, that asks "do these pages answer the question?" Example: it says "no, nothing about contractors".
Query rewriting
Turning a question into a better search. Example: "Do contractors get it?" becomes "contractor eligibility for parental leave".
Sub-question
One small piece of a bigger question. Example: "What changed in the leave policy?" is half of the manager's question.
Step cap
A hard limit on how many loops are allowed. Example: at most 3 searches, then stop.
Log (trace)
A saved record of every step the agent took. Example: "search 2: handbook, query X, graded not enough".
Tool
Something the model can call to do a job. Example: a search tool that looks in the handbook.
Model call
One request sent to the AI model. Example: one grading check is one model call.
Prompt
The written instructions you give the model. Example: "Answer only from these pages."
Chunk
A small piece of a document, cut for search. Example: one page of the handbook.
Sample
A small set of real cases you check by hand. Example: 50 real questions.
Citation
A named source for a claim, so a reader can check it. Example: "Contractor handbook, page 4".

Related ideas you may hear: corrective RAG and self-checking RAG. They focus on the grade-and-retry part.

Next: first, what changes when the model gets to choose, and the picture that shows it.

Step 1: Give the assistant a loop, not a single shot

Think of a research assistant. Basic RAG is an assistant who checks one shelf and answers. Agentic RAG is one who can look, think, and look again.

The assistant gets these new powers:

What you are looking at: the loop for the manager's question. The top row is the first try. The bottom row is what happens when the pages are not good enough. Press next to walk along.

On a phone, swipe the picture sideways to see all of it. Grade and Answer sit on the right. Tap Next and swipe to follow each step.

The agentic RAG loop A question goes to a decide step, then a pick-a-source step, then a search step, then a grade step. If the grade says yes, the assistant answers with sources. If the grade says no, it rewrites the question and searches again. If it reaches the step cap, it says it could not find the answer. Question Decidesearch at all? Pick a sourcewhich shelf Search Gradedo pages answer it? Rewrite, retrynew words or source Cap: 3 tries Answerwith sources Say so politely"I could not find it" No Yes cap hit
  1. "Does the new parental leave apply to contractors, and what changed since last year?" It has two parts, so one search may not cover both.
  2. The model asks itself if it needs a lookup. A greeting does not. A company policy does. Why: skipping needless searches saves time and money.
  3. This is routing. Staff leave rules live in the handbook. Contractor terms live in a separate contractor guide. The model picks one to try first.
  4. The chosen source is searched, the same way as basic RAG. Why: the loop is built on top of retrieval, not instead of it.
  5. The grader reads the question and the pages. It asks, "Is the answer here?" Why: this is the step a single-shot system skips.
  6. The pages say nothing about contractors. The model rewrites the search ("contractor eligibility for parental leave"). Then it picks a source again, here the contractor guide. A retry may keep the same source, but a new one often helps. Each trip around the loop counts toward the cap.
  7. Good pages found. The model answers from them and names each page it used, so a person can check.
  8. The cap is 3 tries, a common starting point. Tune it on your own questions. When the tries are used up, the model stops. "I could not find this. Here is what I did find." Why: an honest gap beats a confident guess.
QuestionModel decisionsSearchGives up

Where the picture breaks down:

The grader is not magic. It is one more model call, with a short, strict prompt. Here is a realistic one, in the shape teams actually use:

You are checking whether some retrieved pages answer a question.

Question: {the question}

Pages:
{the pages the search found}

Reply with one verdict only:
ENOUGH — the pages contain everything needed to answer the question.
NOT ENOUGH — something is missing. Name what is missing in one short line.

Do not answer the question yourself. Judge only from the pages above.

Notice what the prompt forbids: answering the question. A grader that answers instead of judging hides the very gap it exists to catch.

Next: you have seen the loop. Now make the choices yourself, as Kavish, and watch the quality and the cost move.

Step 2: Try it: you are Kavish, with the loop in your hands

Why this exercise: to feel the trade-off. More loops can raise answer quality, and they always raise cost and wait.

The rules of this role-play:

Do this (press "Try again" at the end to replay):

  1. Play once the careless way: one search, then ship it. Note the meters.
  2. Play again. This time add a grader, and rewrite when it says "not enough".
  3. Look at the turn after the grader says "not enough". Find the choice that spends more and gets nothing back.
  4. Find the ending that says "I could not find it". It comes after you repeat a search. Decide if that is good or bad. "What to notice" answers it.
What to notice

Next: the role-play used cost as a meter. Let us do the worked arithmetic with made-up numbers, so you can talk about cost with a customer.

Step 3: Count what the extra loops cost

Every box in the loop is a search or a model call. So every loop adds wait and money.

To keep the sums simple, we count only search, grade and answer. We ignore the decide, route and rewrite calls. They add a little more.

These numbers are made up. Check your own tools for real ones.

  1. One search takes 0.3 seconds and costs 0.0005 dollars.
  2. One grading call takes 1.0 second and costs 0.001 dollars.
  3. The final answer call takes 1.5 seconds and costs 0.002 dollars.

Basic RAG (one search, no grading, one answer):

  1. Time: 0.3 + 1.5 = 1.8 seconds.
  2. Cost: 0.0005 + 0.002 = 0.0025 dollars.

Agentic, 1 loop (search and grade once, then answer):

  1. One loop: 0.3 + 1.0 = 1.3 seconds, and 0.0005 + 0.001 = 0.0015 dollars.
  2. Time: 1.3 + 1.5 = 2.8 seconds.
  3. Cost: 0.0015 + 0.002 = 0.0035 dollars.

Agentic, 3 loops (the cap is reached):

  1. Three loops: 3 × 1.3 = 3.9 seconds, and 3 × 0.0015 = 0.0045 dollars.
  2. Time: 3.9 + 1.5 = 5.4 seconds.
  3. Cost: 0.0045 + 0.002 = 0.0065 dollars.

Now compare. Time: 5.4 ÷ 1.8 = 3 times slower. Cost: 0.0065 ÷ 0.0025 = 2.6 times costlier.

Scale it up. At 10,000 questions a day (an example volume, made up):

So the answer is: in this made-up case, a full three-loop run waits 5.4 seconds and costs 2.6 times more. Many questions may need fewer loops, so the real average may sit in between. Measure yours.

Next: cost is the visible risk. The loop has quieter ways to fail, and some questions should never use it.

Step 4: How the loop breaks, and when to skip it

Four ways it fails:

Here is what one logged run looks like, for the manager's question from the start of this chapter. The cap is 3 tries. This run needed 2:

question: "Does the new parental leave apply to contractors, and what changed since last year?"
loop 1 of 3 — source: staff handbook — query: "parental leave policy changes"
  pages found: "Parental leave" (page 12), "Leave types" (page 9), "Returning to work" (page 15)
  grade: NOT ENOUGH — the pages say what changed (12 to 16 weeks) but say nothing about contractors
loop 2 of 3 — rewrite: "contractor eligibility for parental leave" — source: contractor guide
  pages found: "Contractor handbook" (page 4): contractors are not covered by the staff leave policy
  grade: ENOUGH — both parts of the question are now covered
answer: "Parental leave for staff rose from 12 to 16 weeks. Contractors are not covered by the staff policy; a different process applies." (Sources: staff handbook, page 12; contractor handbook, page 4)

With a log like this, a bad answer stops being a mystery. You can point at the exact loop where it went wrong.

When not to use it:

A useful middle path: run one search, grade it, and only loop if the grade says "not enough". Easy questions then stay fast.

Scenario: a customer asks for "an agent for everything" Tomas runs support for a bike-shop chain (a made-up customer). He asks Kavish for an agent that "just figures it out". Tomas keeps fast answers where they matter, and spends the extra calls where they pay.

Next: two quick checks and a prediction, then common mix-ups, and the final four-question quiz.

The cap is a dial. Low is cheap and fast, but hard questions lose their second chance. Pick it by testing on real questions.

Mix-ups to avoid

Recap

In the field A consultant (a composite example) reviews an agent that a team built in a hurry. Add three things before launch: a step cap, a saved log of every step, and a tool-level permission check. Those make an agent safe to run, and easy to fix when it surprises you.
The payoff: the same question, two ways

One search, no loop: "Yes, the new parental leave applies to contractors, and it rose from 12 to 16 weeks." Confident, calm, and wrong. The pages behind it never mentioned contractors.

With the loop: "Parental leave for staff rose from 12 to 16 weeks (staff handbook, 'Parental leave'). Contractors are not covered by the staff policy; a different process applies (contractor handbook, page 4)." Slower, and a reader can check every claim.

One limit to keep in mind. The loop can judge whether it has enough pages. It cannot judge whether its final answer is actually right: a smooth answer can still misread good pages. Checking the answers themselves is an evaluation problem, and later chapters in this course teach it.

Next: the next chapter, "Knowledge graphs and GraphRAG: when relationships matter". A loop helps when one search is not enough. Some questions ask how things connect, and a graph is built for that. The rest of the module also shows how to choose among all these tools, and how to measure whether any of them helped.

Review this chapter in 5 minutes

Watch the whole chapter recapped: the loop, the grader, what the extra tries cost, and when to skip it.

Chapter quiz

Four questions. Score 70% or more to mark this chapter complete.

Review