Deployed Lab

Module 2 · AI engineering foundations · Chapter 7 · Free

Tokens and context windows

12 min1 tokenizer1 budget lab6 checks
In 30 seconds The one picture to remember: the model works at a small desk, and everything must fit on it.
The problem Kavish is an engineer who sets up AI tools at customer sites. A customer asks: "Can your assistant read our 400 contracts, one page each?" He says yes. The first request fails with a "too long" error, and after he sends a smaller slice, the bill is ten times his quote. (A made-up example, but teams meet it all the time.)

Next: to see why Kavish's bill exploded, you need two ideas: tokens (what the model reads and charges for) and the context window (how much fits at once). Then we come back to him.

Words you'll meet

What is a token?

Picture: tokens are like Lego bricks. Text is built from small standard pieces.

Where the picture breaks down: real bricks are all the same size, but tokens vary in length.

How does the tokenizer decide where to cut?

It does not do maths on each word. It looks pieces up in a fixed list.

  1. Before the model is trained, people count which groups of letters show up most often in a huge pile of text.
  2. The common groups go on a list of pieces (commonly tens of thousands of them). A very common word like "the" becomes one piece.
  3. A rare word is not on the list, so it is built from smaller pieces that are.
  4. To cut your text, the tokenizer walks along it and takes the longest piece from the list that fits.

This is often done with a method called byte pair encoding. The exact list differs for each model, so one text has different token counts in different models.

What if the same word appears twice? It becomes the same pieces twice. Both copies count, and you pay for both. Nothing is skipped because it is a repeat.

Try it: watch text turn into tokens

The tool below is a stand-in for a real tokenizer. It does not have the big list. It follows four simple rules instead:

Click the four examples one by one. Then type your own text. Each coloured block is one token.

What to notice: the sentence has 6 words but 7 tokens, the long number costs 4 tokens, and repeating a word repeats its cost. Real tokenizers cut in other places, but the pattern is the same: long, rare and number-heavy text costs more.

Use the AI company's own counter when the count matters.

Next: tokens are the unit everything is counted in. The next question is: how many tokens can the model hold at once?

The context window is a desk

The context window is the size of the model's desk, counted in tokens.

Why is there a limit? The model compares every token with the others as it reads. More tokens means more memory and more work, so builders choose a size they can run. Bigger windows exist, but they cost more to use.

These must all fit on it at once:

Too much, and the request fails with a length error.

The context window as a desk A desk of 8,000 tokens holds five sheets: system prompt 500 tokens, history 2,000, documents 4,200, question 300, and an empty sheet of 1,000 kept for the answer. A sixth sheet does not fit and hangs off the edge. The numbers are made up. Everything has to fit on the desk at once Systemprompt500 tokens Historychat so far2,000 tokens Documentsyou added4,200 tokens Yourquestion300 tokens Answerleft empty1,000 tokens Too muchfalls off Window = 8,000 tokens = 500 + 2,000 + 4,200 + 300 + 1,000 (made-up numbers)
One request, one desk. The model's own answer needs room on the desk too.

Where the picture breaks down: a real desk keeps your papers overnight. The model keeps nothing between requests, so the app must fill the desk again every time.

Nobody hands you a bill that says which sheet used how much. You only see the total, and the total does not tell you what to cut. So builders track each category on its own.

Try it: spend the window

You are the builder. You have a 32,000-token desk (a practice size) and five sheets to put on it.

  1. Drag each slider. The number beside it shows its share of the whole desk.
  2. Push Retrieved documents up to 25,000 and watch the bar go over budget.
  3. Then cut the biggest part, and see the bar fit again.

What to notice: your question is only about 1% of the desk. Documents and history fill it fast. The line under the sliders names the biggest part, which is where you cut first.

Past the window, you must trim something.

Optional: a quick check without the sliders
  1. Start with the sliders' defaults: 1,500 + 6,000 + 12,000 + 300 + 2,000.
  2. Add them: 1,500 + 6,000 = 7,500. Then 7,500 + 12,000 = 19,500.
  3. Then 19,500 + 300 = 19,800. Then 19,800 + 2,000 = 21,800 tokens.
  4. The window is 32,000, so 32,000 - 21,800 = 10,200 tokens are free.
  5. So the answer is: it fits, with 10,200 tokens to spare.

Now drag Documents up to 20,000. The total becomes 29,800, which still fits, but only just.

Back to Kavish: why long chats cost more

This is the first half of her problem: the surprise bill.

Message 50 costs about 19 times message 1, because every old turn is sent again.

Let's work it out. All numbers are made up.

  1. The system prompt is 500 tokens.
  2. Each turn adds 200 tokens: a 40-token message plus a 160-token reply.
  3. Message 1 sends 500 + 40 = 540 tokens.
  4. Before message 50 there are 49 old turns: 49 x 200 = 9,800 tokens.
  5. Message 50 sends 500 + 9,800 + 40 = 10,340 tokens.
  6. Compare: 10,340 / 540 = about 19.
  7. So the answer is: the same short question sends about 19 times more input.
Input size grows with each message in a chat Bars show the input tokens sent: 540 for message 1, 2,340 for message 10, 5,340 for message 25 and 10,340 for message 50. The numbers are illustrative. Tokens sent to the model with each message (made-up chatbot) Message 1 540 tokens Message 10 2,340 tokens Message 25 5,340 tokens Message 50 10,340 tokens
The customer types about the same amount each time. The bar grows because the history is sent again.

This is where Kavish's bill came from. He priced one short message, but real chats carry their history.

Now the second half of her problem: the "too long" error.

Replies often cost more per token than input. Check the AI company's price list.

Scenario: Kavish's 400 contracts
  1. Each contract is one page, about 700 tokens (500 words, from the rule above).
  2. 400 x 700 = 280,000 tokens.
  3. The window is 32,000 tokens: 280,000 / 32,000 = 8.75, so about nine desks.
  4. So the answer is: it can never fit in one request.

The fix is retrieval: search first, then paste only the 3 best paragraphs (Chapter 9).

Her fix was retrieval. A bigger window would not fix the 400 contracts either, because cost and wait grow with it. Step through what grows.

What grows when you send more context Sending more tokens raises cost and waiting time. The fix is to send less, but better. More tokenssent per request Higher bill Longer wait Send less, bettersearch, trim
  1. Pasting everything in "just in case" puts more sheets on the desk than the question needs.
  2. You pay per token on every request, so a big history is paid for again each turn.
  3. More input means more work before the first word comes back, and customers feel that wait.
  4. Search for only the relevant passages, and replace old history with a short summary.
InputRemedy

You decide on purpose what stays on the desk. A bigger window only delays the problem and raises cost and wait time. Chapter 14 (memory systems) goes deeper.

Recap in plain words

In the field A team sized their budget from the average question. They forgot that history is re-sent each turn.

Mix-ups to avoid

Next: Chapter 8 shows how computers find text by meaning, which is how you pick the right pieces to put on the desk.

Chapter quiz

Four questions. Score 70% or more to mark this chapter complete.

Review