Module 3 · Inference engineering · Chapter 37 · Free
GPUs and memory: sizing a model to hardware
- An AI model must fit inside the memory of its GPU to run well.
- Memory needed is the model, plus its chat notes, plus a little spare.
- Simple multiplication tells you the size before you buy anything.
The one picture to remember: GPU memory is a suitcase. If the packing is bigger than the case, the lid will not close.
- Meet the suitcase: the three things that fill GPU memory.
- Look inside a GPU server, to see why memory comes first.
- Do the sums by hand, slowly, for one model.
- Check your sums in a calculator, and change one number at a time.
- Shrink the weights with fewer bits.
- Test yourself with two quick checks and two scenarios, then avoid the usual mix-ups.
After this chapter you can:
- Say why a 16 GB model file needs more than 16 GB of GPU memory.
- Work out weights, chat notes and spare space on paper.
- Tell a customer if their model fits one GPU, and what to change if it does not.
Next: a few words you will see all the way through. Each has a plain meaning and a tiny example.
Words you'll meet
- GPU
- The chip that runs AI models. It does very many sums at the same time. Example: the card in a server rack.
- VRAM
- The memory built into the GPU, measured in gigabytes (GB). It is the suitcase. Example: a card with 24 GB of VRAM.
- Parameter and weights
- A parameter is one number the model learned. All of them together are the weights, which are the model itself. Example: an "8B" model has 8 billion parameters.
- Token
- A small piece of text, about a word or part of one. Example: "refund" is one token, and "unbelievable" may be three.
- Context
- The text the model can see in one chat: your questions and its answers so far. Example: a context of 4,096 tokens is a few pages.
- KV cache
- The model's saved notes about the chat so far. They save it from re-reading everything for each new word. Example: a long chat means big notes.
- Batch size
- How many chats the GPU serves at once. Example: batch size 8 means eight people chatting in parallel.
- Precision
- How many bytes each stored number uses. More bytes means finer detail and more space. A bit is the smallest unit of storage, and 8 bits make 1 byte. Example: 16-bit is 2 bytes, 8-bit is 1 byte.
- Quantization
- Storing each number in fewer bits to save space. Example: from 16-bit to 4-bit makes the weights four times smaller.
- Overhead
- Extra memory lost to gaps, buffers and housekeeping. Example: we plan 15% on top (a common guess, not a law).
- Layer, attention head, KV head
- A layer is one stage the text passes through. Each layer has many attention heads, which are small parts that each look at the text in their own way. A KV head is a set of notes that several attention heads can share. Fewer KV heads means a smaller cache. Example: 64 attention heads may share only 8 KV heads. Head size is how many numbers each head holds.
- Mixture of experts
- A model built from many specialist parts, where each word uses only a few. Example: 100 billion total parameters, but only some used per word. All the parts still sit in memory.
Next: back to Kavish. He needs a way to picture memory before he can count it. We use a suitcase.
Step 1: Meet the suitcase: three things fill GPU memory
Ask a question and get a long answer: the model must remember all of it. So it keeps notes. Those notes take memory too.
GPU memory, called VRAM, is a suitcase. Three things go inside. Click through the steps below.
- Heavy clothes are the weights: all the learned numbers. Packed for every chat.
- Souvenirs are the KV cache: the chat notes. They pile up.
- Lost space is overhead: odd gaps and buffers.
- Memory is counted in gigabytes (GB). A common graphics card has 24 GB.
On a phone, swipe a picture sideways to see all of it. Click "Next" under it to step through.
What you are looking at: one suitcase cut into three parts. The widths are only for illustration. The real shares for one example come in Step 3.
- GPU memory is measured in gigabytes. It does not stretch. Everything the model needs must fit inside it.
- The weights are the model itself. They are packed once and stay in place. They usually take the biggest part, and the same space in every chat.
- Every chat adds notes. Longer chats and more people mean more souvenirs. This part is the one that surprises people.
- Gaps and buffers waste a little room. We plan for it with a percentage on top.
- Add the three parts. If the total is bigger than the suitcase, the model will not run well. Fit is a yes or no.
Where the picture breaks down:
- Same: the case has a fixed size, and the parts compete for it.
- Different: souvenirs are freed when a chat ends. A real suitcase keeps them.
- So pack for the busiest moment, not the average one.
Next: the suitcase is a picture. Let us open a real GPU server and see where weights, notes and the sums live. That shows why memory comes first.
Step 2: Inside a GPU server: why memory comes first
A GPU server has a few parts. Four steps explain the job of each. The "host" is the ordinary computer that holds the GPU. Speed topics such as bandwidth (how fast data moves) come in later chapters.
What you are looking at: four boxes left to right. Arrows show where the data moves. Click through.
- The host copies the weights into GPU memory when the server starts. After that they stay there. If they do not fit, the model cannot run well.
- The notes (the KV cache) sit in the same memory. They grow with longer chats and more people. Now both boxes of the suitcase live in this one place.
- For every new word, the compute units read the weights and notes from GPU memory. That is why memory must sit close by.
- If one card is too small, a fast link lets two GPUs share one model. Later chapters cover how.
- Capacity (GB) is the suitcase size. It decides what fits.
- Fit is yes or no, so check it first.
- So the first question is always: does it fit in memory?
Next: now we count. We take one made-up model and work out its memory line by line. Then the calculator will make sense.
Step 3: Do the sums by hand, slowly
A made-up 8-billion-parameter model. Numbers are illustrative. Here is the setup.
Try it yourself: cover the answers and try sum 1 before you read it. Then check each next sum.
- 2 bytes per number (16-bit).
- Chats of 4,096 tokens each (a token is a word piece).
- 8 chats open at once.
- Model has 32 layers, 8 KV heads, head size 128 (made-up, but typical in size).
- We compare with a 24 GB card, a common graphics card size.
- 1 GB is 1 billion bytes.
- Overhead is 15% (a common planning guess, not a law).
Why the notes formula looks long:
- For each token, each layer stores one key and one value.
- Each is stored for every KV head.
- Each is a list of "head size" numbers.
- Weights: 8 billion x 2 bytes = 16 billion bytes = 16 GB.
- Notes for one token: 2 x 32 layers x 8 heads x 128 size x 2 bytes = 131,072 bytes. (The first 2 is one key plus one value. 2 x 32 = 64, x 8 = 512, x 128 = 65,536, x 2 bytes = 131,072.)
- Notes for all chats: 131,072 x 4,096 tokens x 8 chats = about 4.29 billion bytes = 4.29 GB.
- Overhead: 15% x (16 + 4.29) = about 3.04 GB.
- Add up: 16 + 4.29 + 3.04 = about 23.3 GB.
So the answer is about 23.3 GB. A 24 GB GPU fits it, with about 0.7 GB spare.
What you are looking at: the same three parts as a bar, drawn to scale inside a 24 GB GPU. Click through.
- 8 billion numbers at 2 bytes each make 16 GB. This part is fixed. It does not change with users.
- 8 chats of 4,096 tokens each add 4.29 GB. Double the chats and this block doubles.
- We add 15% of weights plus notes, about 3.04 GB. It covers gaps and buffers.
- The total is about 23.3 GB, inside 24 GB. It fits, but almost no room is left.
Next: you did one case by hand. A calculator lets you change any number and watch the total move. It uses the same sums.
Step 4: Try it: the VRAM calculator
Why this exercise: to feel which number moves the total most, before you promise a customer anything.
The rules it follows:
- Weights = parameters x bytes per number.
- Notes = 2 x layers x KV heads x head size x bytes x tokens x chats.
- Overhead = a percentage of weights plus notes.
- Total = the three added together. 1 GB is 1 billion bytes.
The "Fits?" table compares the total with a card. For example, 8 GPUs x 80 GB is one big server with 640 GB combined.
Its limits: it is an estimate. Real engines add a few extras. The presets are made up and are not real model specs.
Do this:
- Press "Small model, short chats". It is the same setup as your hand sum. The total should match your hand sum, about 23.3 GB. The 24 GB row says it fits.
- Change "Batch size" (the last box in the top grid) from 8 to 16. Watch the KV cache double and the 24 GB row flip to "Does not fit".
- Press "Large model, many users". It uses 8-bit weights, so the weights are 70 GB. Look at the coloured bar: the cache bar should be biggest.
- The "Mid-size" and "Tiny" presets are optional extras: try them to see long chats or one user.
- Open "Show the formulas with your numbers" to see each line worked out.
VRAM calculator (illustrative presets)
Want it as a single file? Open source on GitHub (MIT licence).
Advanced (already filled in)
| GPU memory each (GB) | GPUs | Combined | Fits? |
|---|
Show the formulas with your numbers
- At batch 16 the 24 GB row flips to "Does not fit". The weights never change, but the cache grows with every extra person.
- In "Large model, many users" the cache is bigger than the weights (about 86 GB against 70 GB). The calculator says so in the box under the bar.
- The "Fits?" table shows what a bigger or extra GPU buys you.
Next: the weights were the biggest block for most presets. Can we make them smaller? Yes: store each number in fewer bits.
Step 5: Make the weights smaller with fewer bits
Remember: 8 bits make 1 byte. So 16-bit is 2 bytes and 4-bit is half a byte.
Think of rounding prices to the nearest dollar. The list gets shorter to write down. Storing numbers in fewer bits is called quantization.
What you are looking at: the same 8-billion-parameter model stored four ways. Bar length is memory for the weights. Click through.
- 8 billion x 4 bytes = 32 GB. Very fine detail, and a very big suitcase.
- 8 billion x 2 bytes = 16 GB. This is the usual starting point in this chapter.
- 8 billion x 1 byte = 8 GB. Half the space of 16-bit.
- 8 billion x 0.5 byte = about 4 GB. The 16 GB weights shrink to about 4 GB.
Back to Kavish: with 8-bit weights, the 16 GB model needs 8 GB. On the 24 GB card, that leaves about 8 GB spare for cache and overhead.
Where the picture breaks down:
- Same: fewer digits means a shorter list.
- Different: rounding can nudge answers off a little. Prices rounded to the dollar still add up. A model's answers may not.
- So test answer quality after you shrink.
Next: time to test your sums. Two quick checks, then two customer cases.
Step 6: Test yourself, then meet two customer cases
Why these checks: weights and cache are the two numbers you will compute most often.
Two quick checks
A customer scenario: will it fit?
Kavish's hospital group asks for a big model on one card. Make your guess before you read the answer.
70 billion x 2 bytes = 140 GB for the weights alone. At 4 bits, about 35 GB for weights. Add cache and overhead, so check with the calculator: set 70 B and 4-bit. Cards come in sizes such as 24, 48 and 80 GB. Then test answer quality, for example by running the same sample questions before and after.
A second scenario: more users arrive
Now the worked-example model is live. Staff like it, and more people join.
Weights stay 16 GB. Cache doubles to about 8.59 GB. Overhead is about 3.69 GB. Total is about 28.3 GB, which is over 24 GB. Fix: a smaller batch, shorter chats, or 8-bit weights.
Next: a few wrong ideas that catch newcomers. Then a short recap.
Mix-ups to avoid
- "The file size is the memory needed." No. The file is only the weights. Notes and overhead come on top.
- "More users only means more speed needed." No. Each user also adds notes, so memory grows too.
- "If it fits, it will be fast." No. Fit is yes or no. Speed depends on other things, such as memory bandwidth (how fast data moves) and compute (Chapter 36). Bandwidth means how fast data moves between memory and compute.
Recap in plain words
- Memory needed = weights + KV cache + overhead.
- Weights = parameters x bytes each. Fewer bits shrink them.
- Size for the busiest moment, because chats fill the cache.
- Fit is yes or no. A model that fits can still be slow.
- The weights fit.
- The cache for twenty long chats did not.
- Asking about chat length and users first gave real options in minutes.
Next: Kavish now knows what fits. Chapter 38 asks what software puts the model on the GPU: CUDA, frameworks and inference engines. Quiz first.
Chapter quiz
Four questions. Score 70% or more to mark this chapter complete.