Deployed Lab

Module 3 · Inference engineering · Chapter 37 · Free

GPUs and memory: sizing a model to hardware

10 min4 clickable diagrams1 VRAM calculator2 scenarios2 quick checksfinal quiz
In 30 seconds

The one picture to remember: GPU memory is a suitcase. If the packing is bigger than the case, the lid will not close.

The problem A composite example: Kavish buys hardware for a hospital group. Patient questions cannot leave the building, so he needs his own GPUs. His first question is how much GPU memory to buy. The surprise: a model whose files are 16 GB needs about 23 GB to run.
The plan for this chapter
  1. Meet the suitcase: the three things that fill GPU memory.
  2. Look inside a GPU server, to see why memory comes first.
  3. Do the sums by hand, slowly, for one model.
  4. Check your sums in a calculator, and change one number at a time.
  5. Shrink the weights with fewer bits.
  6. Test yourself with two quick checks and two scenarios, then avoid the usual mix-ups.

After this chapter you can:

Next: a few words you will see all the way through. Each has a plain meaning and a tiny example.

Words you'll meet

GPU
The chip that runs AI models. It does very many sums at the same time. Example: the card in a server rack.
VRAM
The memory built into the GPU, measured in gigabytes (GB). It is the suitcase. Example: a card with 24 GB of VRAM.
Parameter and weights
A parameter is one number the model learned. All of them together are the weights, which are the model itself. Example: an "8B" model has 8 billion parameters.
Token
A small piece of text, about a word or part of one. Example: "refund" is one token, and "unbelievable" may be three.
Context
The text the model can see in one chat: your questions and its answers so far. Example: a context of 4,096 tokens is a few pages.
KV cache
The model's saved notes about the chat so far. They save it from re-reading everything for each new word. Example: a long chat means big notes.
Batch size
How many chats the GPU serves at once. Example: batch size 8 means eight people chatting in parallel.
Precision
How many bytes each stored number uses. More bytes means finer detail and more space. A bit is the smallest unit of storage, and 8 bits make 1 byte. Example: 16-bit is 2 bytes, 8-bit is 1 byte.
Quantization
Storing each number in fewer bits to save space. Example: from 16-bit to 4-bit makes the weights four times smaller.
Overhead
Extra memory lost to gaps, buffers and housekeeping. Example: we plan 15% on top (a common guess, not a law).
Layer, attention head, KV head
A layer is one stage the text passes through. Each layer has many attention heads, which are small parts that each look at the text in their own way. A KV head is a set of notes that several attention heads can share. Fewer KV heads means a smaller cache. Example: 64 attention heads may share only 8 KV heads. Head size is how many numbers each head holds.
Mixture of experts
A model built from many specialist parts, where each word uses only a few. Example: 100 billion total parameters, but only some used per word. All the parts still sit in memory.

Next: back to Kavish. He needs a way to picture memory before he can count it. We use a suitcase.

Step 1: Meet the suitcase: three things fill GPU memory

Ask a question and get a long answer: the model must remember all of it. So it keeps notes. Those notes take memory too.

GPU memory, called VRAM, is a suitcase. Three things go inside. Click through the steps below.

On a phone, swipe a picture sideways to see all of it. Click "Next" under it to step through.

What you are looking at: one suitcase cut into three parts. The widths are only for illustration. The real shares for one example come in Step 3.

GPU memory as a suitcase A suitcase split into three parts. The largest part is heavy clothes, which stand for the model weights. The middle part is souvenirs, which stand for the chat memory that grows during the trip. The last part is lost packing space, which stands for overhead. Suitcase size = GPU memory (GB) Heavy clothes= model weights Souvenirs= KV cache Lost space= overhead Always packed.Same size every chat. Grows as chatsget longer.More people too. Odd gapsand buffers. If the total is biggerthan the suitcase,the lid will not close.
  1. GPU memory is measured in gigabytes. It does not stretch. Everything the model needs must fit inside it.
  2. The weights are the model itself. They are packed once and stay in place. They usually take the biggest part, and the same space in every chat.
  3. Every chat adds notes. Longer chats and more people mean more souvenirs. This part is the one that surprises people.
  4. Gaps and buffers waste a little room. We plan for it with a percentage on top.
  5. Add the three parts. If the total is bigger than the suitcase, the model will not run well. Fit is a yes or no.
WeightsKV cacheOverhead

Where the picture breaks down:

Next: the suitcase is a picture. Let us open a real GPU server and see where weights, notes and the sums live. That shows why memory comes first.

Step 2: Inside a GPU server: why memory comes first

A GPU server has a few parts. Four steps explain the job of each. The "host" is the ordinary computer that holds the GPU. Speed topics such as bandwidth (how fast data moves) come in later chapters.

What you are looking at: four boxes left to right. Arrows show where the data moves. Click through.

What is inside a GPU server for inference A host computer with ordinary memory connects to a GPU whose memory holds weights and cache. The GPU memory feeds the compute units. A fast link connects this GPU to a second GPU for splitting a large model. Hostregular memory GPU memoryweights + notes Computedoes the sums Second GPUshares the loadown memory copies weightsonce reads weights andnotes for every word fast link between the two GPUs
  1. The host copies the weights into GPU memory when the server starts. After that they stay there. If they do not fit, the model cannot run well.
  2. The notes (the KV cache) sit in the same memory. They grow with longer chats and more people. Now both boxes of the suitcase live in this one place.
  3. For every new word, the compute units read the weights and notes from GPU memory. That is why memory must sit close by.
  4. If one card is too small, a fast link lets two GPUs share one model. Later chapters cover how.
HostGPU memoryComputeSecond GPU

Next: now we count. We take one made-up model and work out its memory line by line. Then the calculator will make sense.

Step 3: Do the sums by hand, slowly

A made-up 8-billion-parameter model. Numbers are illustrative. Here is the setup.

Try it yourself: cover the answers and try sum 1 before you read it. Then check each next sum.

Why the notes formula looks long:

  1. Weights: 8 billion x 2 bytes = 16 billion bytes = 16 GB.
  2. Notes for one token: 2 x 32 layers x 8 heads x 128 size x 2 bytes = 131,072 bytes. (The first 2 is one key plus one value. 2 x 32 = 64, x 8 = 512, x 128 = 65,536, x 2 bytes = 131,072.)
  3. Notes for all chats: 131,072 x 4,096 tokens x 8 chats = about 4.29 billion bytes = 4.29 GB.
  4. Overhead: 15% x (16 + 4.29) = about 3.04 GB.
  5. Add up: 16 + 4.29 + 3.04 = about 23.3 GB.

So the answer is about 23.3 GB. A 24 GB GPU fits it, with about 0.7 GB spare.

What you are looking at: the same three parts as a bar, drawn to scale inside a 24 GB GPU. Click through.

Memory needed for the worked example, drawn to scale A bar split into three parts: weights 16 GB, KV cache 4.29 GB, overhead 3.04 GB, totalling about 23.3 GB. It sits inside an outline of a 24 GB GPU, with a little space left over. One 24 GB GPU (illustrative size) Weights: 16 GB8 billion x 2 bytes KV cache4.29 GB Overhead3.04 GB Total needed: about 23.3 GB It fits, with only about 0.7 GB to spare. Very tight, so add a few more users and it breaks. Bar widths are to scale.
  1. 8 billion numbers at 2 bytes each make 16 GB. This part is fixed. It does not change with users.
  2. 8 chats of 4,096 tokens each add 4.29 GB. Double the chats and this block doubles.
  3. We add 15% of weights plus notes, about 3.04 GB. It covers gaps and buffers.
  4. The total is about 23.3 GB, inside 24 GB. It fits, but almost no room is left.
WeightsKV cacheOverhead

Next: you did one case by hand. A calculator lets you change any number and watch the total move. It uses the same sums.

Step 4: Try it: the VRAM calculator

Why this exercise: to feel which number moves the total most, before you promise a customer anything.

The rules it follows:

The "Fits?" table compares the total with a card. For example, 8 GPUs x 80 GB is one big server with 640 GB combined.

Its limits: it is an estimate. Real engines add a few extras. The presets are made up and are not real model specs.

Do this:

  1. Press "Small model, short chats". It is the same setup as your hand sum. The total should match your hand sum, about 23.3 GB. The 24 GB row says it fits.
  2. Change "Batch size" (the last box in the top grid) from 8 to 16. Watch the KV cache double and the 24 GB row flip to "Does not fit".
  3. Press "Large model, many users". It uses 8-bit weights, so the weights are 70 GB. Look at the coloured bar: the cache bar should be biggest.
  4. The "Mid-size" and "Tiny" presets are optional extras: try them to see long chats or one user.
  5. Open "Show the formulas with your numbers" to see each line worked out.

VRAM calculator (illustrative presets)

Want it as a single file? Open source on GitHub (MIT licence).

Advanced (already filled in)
Weights
KV cache
Overhead
Total needed
WeightsKV cacheOverhead
Does it fit? Edit the memory per GPU (GB, illustrative) and the number of GPUs.
GPU memory each (GB)GPUsCombinedFits?
Show the formulas with your numbers
What to notice

Next: the weights were the biggest block for most presets. Can we make them smaller? Yes: store each number in fewer bits.

Step 5: Make the weights smaller with fewer bits

Remember: 8 bits make 1 byte. So 16-bit is 2 bytes and 4-bit is half a byte.

Think of rounding prices to the nearest dollar. The list gets shorter to write down. Storing numbers in fewer bits is called quantization.

What you are looking at: the same 8-billion-parameter model stored four ways. Bar length is memory for the weights. Click through.

The same model stored at four precisions Four bars for an 8-billion-parameter model. At 32-bit the weights need 32 GB. At 16-bit, 16 GB. At 8-bit, 8 GB. At 4-bit, about 4 GB. Each step down halves the memory. 32-bit (4 bytes)32 GB 16-bit (2 bytes)16 GB 8-bit (1 byte)8 GB 4-bit (0.5 byte)4 GB Same model, same 8 billion parameters.
  1. 8 billion x 4 bytes = 32 GB. Very fine detail, and a very big suitcase.
  2. 8 billion x 2 bytes = 16 GB. This is the usual starting point in this chapter.
  3. 8 billion x 1 byte = 8 GB. Half the space of 16-bit.
  4. 8 billion x 0.5 byte = about 4 GB. The 16 GB weights shrink to about 4 GB.
Weights

Back to Kavish: with 8-bit weights, the 16 GB model needs 8 GB. On the 24 GB card, that leaves about 8 GB spare for cache and overhead.

Where the picture breaks down:

Next: time to test your sums. Two quick checks, then two customer cases.

Step 6: Test yourself, then meet two customer cases

Why these checks: weights and cache are the two numbers you will compute most often.

Two quick checks

A customer scenario: will it fit?

Kavish's hospital group asks for a big model on one card. Make your guess before you read the answer.

70 billion x 2 bytes = 140 GB for the weights alone. At 4 bits, about 35 GB for weights. Add cache and overhead, so check with the calculator: set 70 B and 4-bit. Cards come in sizes such as 24, 48 and 80 GB. Then test answer quality, for example by running the same sample questions before and after.

A second scenario: more users arrive

Now the worked-example model is live. Staff like it, and more people join.

Weights stay 16 GB. Cache doubles to about 8.59 GB. Overhead is about 3.69 GB. Total is about 28.3 GB, which is over 24 GB. Fix: a smaller batch, shorter chats, or 8-bit weights.

Next: a few wrong ideas that catch newcomers. Then a short recap.

Mix-ups to avoid

Recap in plain words

In the field A composite example: a customer asked for a yes or no on one GPU.
Go deeper This chapter follows the topic order of the book Inference Engineering by Philip Kiely (Baseten Books): sections 3.1 GPU Architecture, 3.2 GPU Architecture Generations, 3.3 Instances, 3.4 Other Datacenter Accelerator Options and 3.5 Local Inference. The book is at baseten.co/inference-engineering. This course is independent material, not endorsed by the author or Baseten. This chapter covers only memory sizing. GPU generations, instances and local inference are not covered here. Official hardware pages: NVIDIA H100 and NVIDIA Blackwell architecture.

Next: Kavish now knows what fits. Chapter 38 asks what software puts the model on the GPU: CUDA, frameworks and inference engines. Quiz first.

Chapter quiz

Four questions. Score 70% or more to mark this chapter complete.

Review