Module 3 · Inference engineering · Chapter 32 · Free
Why run models yourself? Inference engineering for FDEs
- You can rent an AI model by the word, or run one on your own computers. Running a model to get an answer is called inference.
- Renting costs less when you use it a little. Running your own costs less when you use it a lot.
- Running your own also means you look after it, so count that work as a cost.
The one picture to remember: a taxi (pay per ride) versus owning a car (pay every month, trips almost free).
- One firm rule: claim text may never leave the insurer's own cloud account.
- The pilot cost $1,600 last month.
- Rosa, from finance, asks: "At ten times the use, can we just run the model ourselves?"
- Learn a picture for the choice: a taxi or a car (and a middle option).
- See what "owning the car" really means: three layers of work.
- Walk a decision tree, the way you would in a customer meeting.
- Do the sums, then test your own numbers in a calculator.
- Practise on two customer cases, then check the usual mix-ups.
After this chapter you can:
- Explain in one minute when renting a model costs less, and when running your own does.
- Work out the break-even point from three numbers, and say what the sum leaves out.
- Tell a customer where to start when their data rules or their skills limit the choice.
Next: a few words you will see all the way through. Each has a plain meaning and a small example. The four marked "key" matter most: API, GPU, idle and break-even.
Words you'll meet
- Inference
- Running a model to get an answer. Example: the insurer sends a claim, and the model's reply is one inference.
- Forward deployed engineer (FDE)
- An engineer who works with a customer to build and run AI for them. Example: Kavish, helping the insurer.
- Model
- The AI program that reads text and writes answers. Example: the program that reads a claim and writes a summary.
- Token
- A small piece of text, a bit shorter than a word. Services charge by the token. Example: "refund" may be one token, and "refunded" may be two.
- API (key)
- A doorway that lets your program call someone else's model over the internet. Example: you send a claim, and the answer comes back.
- Open-weight model
- A model whose learned settings are published, so anyone can download and run it. Example: you download it once and run it on your own machines.
- GPU (key)
- A powerful chip that runs models fast. It costs a lot to rent. Example: one rented GPU might cost thousands of dollars a month (made-up number).
- Self-hosting
- Running a model on computers you control. Example: the insurer runs the model in its own cloud account.
- Managed service
- A provider runs the model for you, but inside your own cloud account. Example: the data stays in the insurer's account, and someone else keeps the GPUs running.
- Idle (key)
- Paid for, but not working. Example: a GPU at 3 a.m. with no questions to answer still costs the same per hour.
- Break-even (key)
- The use level where both options cost the same. Example: below it the taxi is cheaper, above it the car is cheaper.
- Eval set
- A small list of test questions with known good answers. Example: 50 old claims with the summary a person wrote for each.
- Weights
- The model's learned settings. Custom weights means a model you trained further on your own examples. Example: a small model taught the insurer's claim wording.
- On-call
- Being the person who fixes it when it breaks, even at night. Example: the model stops answering at 3 a.m., and your phone rings.
Next: back to Kavish. He starts with a simple picture that makes the choice easy to explain.
Step 1: A taxi or a car? The picture for the choice
Think about how you get around a city.
- A taxi charges per ride. You pay nothing on days you stay home.
- A car costs a fixed sum every month. Each extra trip is almost free.
Calling a model through an API is the taxi. You pay per token.
Self-hosting an open-weight model is the car. You rent GPUs (the chips that run the model) and run the model yourself.
What you are looking at: use per month runs along the bottom, in millions of tokens. Cost per month runs up the side, in dollars. The numbers are made up and match the sums in Step 4. Press next to add one line at a time.
- Along the bottom is how much you use the model. Up the side is what you pay each month.
- Every ride adds to the bill. Use nothing and you pay nothing. Use twice as much and you pay twice as much.
- You pay for the GPUs and the people every month, busy or not. Extra trips add almost nothing.
- At this use level both options cost the same. Finding this point is the job of the whole chapter.
- Left of the point, the taxi is cheaper. Right of it, the car is cheaper. So the real question is: which side are we on?
Where the picture breaks down:
- A car parked in the garage costs nothing extra. Idle GPUs still cost money every hour, even at 3 a.m.
- A taxi ride has one price. Tokens are priced per million, and providers change their prices.
There is also a middle option: a managed service in your own cloud account. Think of a rental car with a driver. The data stays with you, and someone else keeps the GPUs running.
Why would a customer choose the car?
- Data rules: Kavish's insurer must keep claim text inside its own cloud account.
- Steady heavy use: a nightly job that keeps GPUs busy makes each token cheaper.
- Control: a provider can stop offering a model your app relies on. Your own copy keeps working. You can also teach a small model your own examples, and only you can run it.
Why this check: to see the taxi rule in a small case. Click one option, and you will see why it is right or wrong.
Next: the car sounds cheap, but "owning" hides a lot of work. Step 2 opens the hood: what must you build and run, and who is on call?
Step 2: What owning the car really means: three layers
With an API, someone else built and runs everything. If you self-host, you own three layers of work.
- Runtime: the model running on a GPU.
- Infrastructure: many copies, shared load, and GPU supply.
- Tooling: releases, watching, tests and cost counting.
What you are looking at: a stack of three boxes. Requests enter at the top and flow down. Press next to meet each layer.
- Real people send questions at different times of day. Everything below exists to answer them.
- This is the model itself, running on a GPU. It sets how fast one copy answers.
- It shares questions between copies, and adds copies when busy. It keeps answers steady when traffic doubles.
- Releases, watching for slow answers, test questions, and cost per request. It shows problems before customers do.
- A great model on weak infrastructure still gives slow, patchy answers. Ask who owns each layer before you promise a self-hosting project.
Owning the car also means on-call: someone must fix it, even at 3 a.m. That person's time is a real cost.
Next: Kavish now knows both options and what the car demands. Step 3 gives him a way to choose, one question at a time.
Step 3: Choose with a decision tree
Why this exercise: to practise the order of questions you would ask in a real customer meeting.
The tree follows four rules, in this order. Data comes first. Skills come next, but only after a hard data rule. Volume comes after.
- Data rules come first. They can decide the answer before cost matters.
- Then skills: who can run GPUs?
- Then volume: high and steady favours the car.
- "Not measured yet" always means: start small and measure.
Its limit: the tree is a first filter. It does not know your real prices or quality results.
Do this:
- Play the insurer first. Click "Yes, a hard rule", then "No" to GPU skills. Read the result.
- Start again as the small helper from the quiz. Click "No, a provider's terms are acceptable", then "Neither, volume is modest or bursty".
- Start again with high steady volume and nothing measured. Click "No, a provider's terms are acceptable", then "Yes, high steady volume or custom weights", then "No, nothing measured yet". Notice that it does not say "buy GPUs".
- At the end of each path, read the "why" line and the "not" line. They are what you would say to the customer.
Why this check: to test the idea that the car pays off only when it is busy. Click one option, and you will see why it is right or wrong.
Next: the tree said "volume matters". But how high is high? Step 4 puts numbers on it, using Kavish's insurer.
Step 4: Do the sums: where is break-even?
Let's work it out together. All numbers are made up for the example. First, the inputs:
| Input | Value | Where it comes from |
|---|---|---|
| API price | $2 per million tokens | The provider's price list |
| Pilot use | 800 million tokens a month | Last month's bill: $1,600 / $2 = 800 |
| GPUs | $4,000 a month for one set | A quote for rented GPUs |
| People | $2,000 a month | Part-time engineer and on-call time |
| One set can serve | 4,000 million tokens a month | A test run on that set |
Now the sums:
- Pilot on the API: 800 x $2 = $1,600 a month. This matches the story.
- Self-hosting: $4,000 + $2,000 = $6,000 a month, used or not.
- Break-even: $6,000 / $2 = 3,000 million tokens a month.
- Why divide? Dollars a month divided by dollars per million tokens gives million tokens a month. It is the use at which the API bill reaches $6,000.
- The pilot uses 800 million. That is far below 3,000.
So the answer is: stay on the API for now.
Now try ten times the use. That is 8,000 million tokens.
- API: 8,000 x $2 = $16,000 a month.
- One set of GPUs serves only 4,000 million. GPUs come in whole sets, so you cannot rent 1.5. You need 2 sets: 2 x $4,000 = $8,000.
- Add $2,000 for people. Self-hosting is now $10,000 a month.
- Compare: $16,000 - $10,000 = $6,000.
So the answer is: at ten times the use, self-hosting saves about $6,000 a month on paper. Check quality and on-call first. Note that the $6,000 break-even used one set. The second set moves break-even up, but 8,000 is still above it.
What you are looking at: the two cases side by side. Each bar is dollars a month. The shortest bar wins.
Next: now try the same sums with your own numbers in the calculator.
Try it: the break-even calculator
Why this exercise: to see how fast the answer changes when you change one number.
The calculator follows one simple rule:
- Fixed cost = GPU cost + engineering cost.
- Break-even = fixed cost / API price per million tokens.
- If your volume is above break-even, self-hosting is cheaper on paper.
Its limits: it does not know about quality, traffic spikes, or how long setup takes. Treat it as a first filter, not a quote.
Do this:
- Leave the numbers as they are. This is the pilot at 800. The bar is short, and the text says the API is cheaper.
- Type 8,000 into the first box. The bar passes the dark line. A caution says you need 2 sets of GPUs, so the real fixed cost is $10,000, not the $6,000 in the first line.
- Set the first box back to 800. Click the word "Advanced" to open it. Set the last box there (millions of tokens the GPUs can serve) to 20,000. The caution now says about 4% use: "low, paying for idle hardware".
- Set the API price to 0.5. Break-even jumps from 3,000 to 12,000, so the bar fill shrinks. A cheaper taxi stays better for longer.
The bar rescales every time, so the dark line can stay in the same spot. Read the numbers and the length of the filled part.
Advanced
Break-even = fixed cost / price. Real life adds idle time, scaling steps and quality, so treat this as a first filter, not a quote.
Next: theory is done. Step 5 gives you two customer cases to practise on.
Step 5: Practise on two customer cases
Each case has a different clue that decides the answer. Click your answer first, and the reasons appear.
Steady volume and a narrow job favour the car. But nothing is measured yet, so measure first. The hybrid answer: self-host the steady part, rent the rest. First step: count your monthly tokens, as in the sum above. About 600 million is well below 3,000, so measure before you buy anything. Prices here are made up, so use your own.
The data rule rules out a public API. No GPU skills rules out raw self-hosting. A managed service in their account fits both.
Next: a short list of wrong ideas that catch real teams, then a recap.
Mix-ups to avoid
- "Open-weight means free." The model costs nothing to download. GPUs and on-call people still cost money.
- "Self-hosting is always cheaper." Only when the GPUs stay busy. Idle GPUs cost the same as busy ones.
- "Pick one for good." Many teams mix both. They self-host the steady load and rent the spikes.
Recap
- An API is a taxi. Self-hosting is a car.
- Break-even is fixed cost divided by price per token.
- Data rules come first. Then skills, then volume.
- Not sure yet? Start on the API and measure.
- They added overnight batch jobs for the same GPUs.
- The GPUs stayed busy around the clock.
- Cost per token dropped.
The lesson: busy GPUs are cheap. Idle GPUs are not.
Next: Kavish can now say "stay on the API for now, and here is the number that changes that". Before he can promise anything about a car, he must measure how fast it answers and how many people it serves at once. That is Chapter 33, Measuring latency and throughput.
Chapter quiz
Click one option for each question. Four questions. Score 70% or more to mark this chapter complete.