Module 2 · AI engineering foundations · Chapter 7 · Free
Tokens and context windows
- A model reads small pieces of text called tokens, not whole words.
- It can hold only a fixed number of tokens at once, and you pay for each one.
- A chat sends its whole history again each turn, so long chats cost far more.
Next: to see why Kavish's bill exploded, you need two ideas: tokens (what the model reads and charges for) and the context window (how much fits at once). Then we come back to him.
Words you'll meet
- Token: a small piece of text the model reads.
- Tokenizer: the tool that cuts text into tokens.
- Context window: the most tokens the model can hold for one request.
- System prompt: hidden rules the app gives the model, like "be polite".
- History: the chat so far, sent again with each new message.
- Input and output: what you send in, and what the model writes back.
- Retrieval: searching first, then sending only the best passages.
What is a token?
- A token is a short word, part of a long word, a piece of a number, or a punctuation mark.
- A tokenizer cuts your text into tokens before the model sees it.
- Rule of thumb for English: one token is about four characters, so a 500-word page is roughly 700 tokens.
- Code, numbers and non-English text need more tokens for the same meaning, so they cost more to send.
Picture: tokens are like Lego bricks. Text is built from small standard pieces.
Where the picture breaks down: real bricks are all the same size, but tokens vary in length.
How does the tokenizer decide where to cut?
It does not do maths on each word. It looks pieces up in a fixed list.
- Before the model is trained, people count which groups of letters show up most often in a huge pile of text.
- The common groups go on a list of pieces (commonly tens of thousands of them). A very common word like "the" becomes one piece.
- A rare word is not on the list, so it is built from smaller pieces that are.
- To cut your text, the tokenizer walks along it and takes the longest piece from the list that fits.
This is often done with a method called byte pair encoding. The exact list differs for each model, so one text has different token counts in different models.
What if the same word appears twice? It becomes the same pieces twice. Both copies count, and you pay for both. Nothing is skipped because it is a repeat.
Try it: watch text turn into tokens
The tool below is a stand-in for a real tokenizer. It does not have the big list. It follows four simple rules instead:
- A word of up to 6 letters is 1 token.
- A longer word is cut into pieces of 4 letters.
- A number is cut into pieces of 3 digits.
- Each punctuation mark is 1 token. A space sticks to the word after it (shown as a dot).
Click the four examples one by one. Then type your own text. Each coloured block is one token.
What to notice: the sentence has 6 words but 7 tokens, the long number costs 4 tokens, and repeating a word repeats its cost. Real tokenizers cut in other places, but the pattern is the same: long, rare and number-heavy text costs more.
Use the AI company's own counter when the count matters.
Next: tokens are the unit everything is counted in. The next question is: how many tokens can the model hold at once?
The context window is a desk
The context window is the size of the model's desk, counted in tokens.
Why is there a limit? The model compares every token with the others as it reads. More tokens means more memory and more work, so builders choose a size they can run. Bigger windows exist, but they cost more to use.
These must all fit on it at once:
- The system prompt
- The chat so far
- Your documents
- Your question
- Room for the answer
Too much, and the request fails with a length error.
Where the picture breaks down: a real desk keeps your papers overnight. The model keeps nothing between requests, so the app must fill the desk again every time.
Nobody hands you a bill that says which sheet used how much. You only see the total, and the total does not tell you what to cut. So builders track each category on its own.
Try it: spend the window
You are the builder. You have a 32,000-token desk (a practice size) and five sheets to put on it.
- Drag each slider. The number beside it shows its share of the whole desk.
- Push Retrieved documents up to 25,000 and watch the bar go over budget.
- Then cut the biggest part, and see the bar fit again.
What to notice: your question is only about 1% of the desk. Documents and history fill it fast. The line under the sliders names the biggest part, which is where you cut first.
Past the window, you must trim something.
Optional: a quick check without the sliders
- Start with the sliders' defaults: 1,500 + 6,000 + 12,000 + 300 + 2,000.
- Add them: 1,500 + 6,000 = 7,500. Then 7,500 + 12,000 = 19,500.
- Then 19,500 + 300 = 19,800. Then 19,800 + 2,000 = 21,800 tokens.
- The window is 32,000, so 32,000 - 21,800 = 10,200 tokens are free.
- So the answer is: it fits, with 10,200 tokens to spare.
Now drag Documents up to 20,000. The total becomes 29,800, which still fits, but only just.
Back to Kavish: why long chats cost more
This is the first half of her problem: the surprise bill.
Message 50 costs about 19 times message 1, because every old turn is sent again.
Let's work it out. All numbers are made up.
- The system prompt is 500 tokens.
- Each turn adds 200 tokens: a 40-token message plus a 160-token reply.
- Message 1 sends 500 + 40 = 540 tokens.
- Before message 50 there are 49 old turns: 49 x 200 = 9,800 tokens.
- Message 50 sends 500 + 9,800 + 40 = 10,340 tokens.
- Compare: 10,340 / 540 = about 19.
- So the answer is: the same short question sends about 19 times more input.
This is where Kavish's bill came from. He priced one short message, but real chats carry their history.
Now the second half of her problem: the "too long" error.
Replies often cost more per token than input. Check the AI company's price list.
- Each contract is one page, about 700 tokens (500 words, from the rule above).
- 400 x 700 = 280,000 tokens.
- The window is 32,000 tokens: 280,000 / 32,000 = 8.75, so about nine desks.
- So the answer is: it can never fit in one request.
The fix is retrieval: search first, then paste only the 3 best paragraphs (Chapter 9).
Her fix was retrieval. A bigger window would not fix the 400 contracts either, because cost and wait grow with it. Step through what grows.
- Pasting everything in "just in case" puts more sheets on the desk than the question needs.
- You pay per token on every request, so a big history is paid for again each turn.
- More input means more work before the first word comes back, and customers feel that wait.
- Search for only the relevant passages, and replace old history with a short summary.
You decide on purpose what stays on the desk. A bigger window only delays the problem and raises cost and wait time. Chapter 14 (memory systems) goes deeper.
Recap in plain words
- Models read tokens, and price, speed and length limits are all counted in tokens.
- Rules, history, documents, question and answer must all fit on one desk.
- A chat re-sends its history, so send less, but better.
- The few very long chats cost the most.
- Always size your budget for those.
Mix-ups to avoid
- "A token is a word." Not quite. Long or odd words split into several tokens.
- "A bigger window fixes everything." It only delays the problem, and costs and waits grow with it.
- "The model remembers our chat." It does not. The app sends the history again each time.
Next: Chapter 8 shows how computers find text by meaning, which is how you pick the right pieces to put on the desk.
Chapter quiz
Four questions. Score 70% or more to mark this chapter complete.