Deployed Lab

Module 3 · Inference engineering · Chapter 32 · Free

Why run models yourself? Inference engineering for FDEs

12 min1 decision tree2 pictures1 break-even calculator4 checks
In 30 seconds

The one picture to remember: a taxi (pay per ride) versus owning a car (pay every month, trips almost free).

The problem Kavish is a forward deployed engineer, or FDE: an engineer who builds AI with a customer, at the customer's side (a made-up example). His customer is an insurer that wants an assistant to read claim files. Kavish needs a clear answer, with numbers, by Friday.
The plan for this chapter
  1. Learn a picture for the choice: a taxi or a car (and a middle option).
  2. See what "owning the car" really means: three layers of work.
  3. Walk a decision tree, the way you would in a customer meeting.
  4. Do the sums, then test your own numbers in a calculator.
  5. Practise on two customer cases, then check the usual mix-ups.

After this chapter you can:

Next: a few words you will see all the way through. Each has a plain meaning and a small example. The four marked "key" matter most: API, GPU, idle and break-even.

Words you'll meet

Inference
Running a model to get an answer. Example: the insurer sends a claim, and the model's reply is one inference.
Forward deployed engineer (FDE)
An engineer who works with a customer to build and run AI for them. Example: Kavish, helping the insurer.
Model
The AI program that reads text and writes answers. Example: the program that reads a claim and writes a summary.
Token
A small piece of text, a bit shorter than a word. Services charge by the token. Example: "refund" may be one token, and "refunded" may be two.
API (key)
A doorway that lets your program call someone else's model over the internet. Example: you send a claim, and the answer comes back.
Open-weight model
A model whose learned settings are published, so anyone can download and run it. Example: you download it once and run it on your own machines.
GPU (key)
A powerful chip that runs models fast. It costs a lot to rent. Example: one rented GPU might cost thousands of dollars a month (made-up number).
Self-hosting
Running a model on computers you control. Example: the insurer runs the model in its own cloud account.
Managed service
A provider runs the model for you, but inside your own cloud account. Example: the data stays in the insurer's account, and someone else keeps the GPUs running.
Idle (key)
Paid for, but not working. Example: a GPU at 3 a.m. with no questions to answer still costs the same per hour.
Break-even (key)
The use level where both options cost the same. Example: below it the taxi is cheaper, above it the car is cheaper.
Eval set
A small list of test questions with known good answers. Example: 50 old claims with the summary a person wrote for each.
Weights
The model's learned settings. Custom weights means a model you trained further on your own examples. Example: a small model taught the insurer's claim wording.
On-call
Being the person who fixes it when it breaks, even at night. Example: the model stops answering at 3 a.m., and your phone rings.

Next: back to Kavish. He starts with a simple picture that makes the choice easy to explain.

Step 1: A taxi or a car? The picture for the choice

Think about how you get around a city.

Calling a model through an API is the taxi. You pay per token.

Self-hosting an open-weight model is the car. You rent GPUs (the chips that run the model) and run the model yourself.

What you are looking at: use per month runs along the bottom, in millions of tokens. Cost per month runs up the side, in dollars. The numbers are made up and match the sums in Step 4. Press next to add one line at a time.

Taxi versus car: how cost grows with use A chart with rides per month along the bottom and monthly cost up the side. The taxi cost climbs steadily with every ride. The car cost is a flat line. The lines cross at a point called break-even. Left of it the taxi is cheaper, right of it the car is cheaper. 0 3,000 $0 $6,000 Use per month (million tokens) Cost per month (dollars) Taxi: pay per ride Car: fixed monthly Break-even point Few rides: taxi wins Many rides: car wins
  1. Along the bottom is how much you use the model. Up the side is what you pay each month.
  2. Every ride adds to the bill. Use nothing and you pay nothing. Use twice as much and you pay twice as much.
  3. You pay for the GPUs and the people every month, busy or not. Extra trips add almost nothing.
  4. At this use level both options cost the same. Finding this point is the job of the whole chapter.
  5. Left of the point, the taxi is cheaper. Right of it, the car is cheaper. So the real question is: which side are we on?
Taxi (API)Car (self-host)Break-evenWho wins

Where the picture breaks down:

There is also a middle option: a managed service in your own cloud account. Think of a rental car with a driver. The data stays with you, and someone else keeps the GPUs running.

Why would a customer choose the car?

Why this check: to see the taxi rule in a small case. Click one option, and you will see why it is right or wrong.

Next: the car sounds cheap, but "owning" hides a lot of work. Step 2 opens the hood: what must you build and run, and who is on call?

Step 2: What owning the car really means: three layers

With an API, someone else built and runs everything. If you self-host, you own three layers of work.

What you are looking at: a stack of three boxes. Requests enter at the top and flow down. Press next to meet each layer.

The three layers of running a model A customer application sends requests down through a tooling layer, an infrastructure layer and a runtime layer that runs the model on GPUs. Users and their requests Layer 3: Toolingreleases, tests, cost Layer 2: Infrastructuremany copies, scaling, GPU supply Layer 1: Runtimethe model running on a GPU
  1. Real people send questions at different times of day. Everything below exists to answer them.
  2. This is the model itself, running on a GPU. It sets how fast one copy answers.
  3. It shares questions between copies, and adds copies when busy. It keeps answers steady when traffic doubles.
  4. Releases, watching for slow answers, test questions, and cost per request. It shows problems before customers do.
  5. A great model on weak infrastructure still gives slow, patchy answers. Ask who owns each layer before you promise a self-hosting project.
UsersToolingInfrastructureRuntime

Owning the car also means on-call: someone must fix it, even at 3 a.m. That person's time is a real cost.

Next: Kavish now knows both options and what the car demands. Step 3 gives him a way to choose, one question at a time.

Step 3: Choose with a decision tree

Why this exercise: to practise the order of questions you would ask in a real customer meeting.

The tree follows four rules, in this order. Data comes first. Skills come next, but only after a hard data rule. Volume comes after.

  1. Data rules come first. They can decide the answer before cost matters.
  2. Then skills: who can run GPUs?
  3. Then volume: high and steady favours the car.
  4. "Not measured yet" always means: start small and measure.

Its limit: the tree is a first filter. It does not know your real prices or quality results.

Do this:

  1. Play the insurer first. Click "Yes, a hard rule", then "No" to GPU skills. Read the result.
  2. Start again as the small helper from the quiz. Click "No, a provider's terms are acceptable", then "Neither, volume is modest or bursty".
  3. Start again with high steady volume and nothing measured. Click "No, a provider's terms are acceptable", then "Yes, high steady volume or custom weights", then "No, nothing measured yet". Notice that it does not say "buy GPUs".
  4. At the end of each path, read the "why" line and the "not" line. They are what you would say to the customer.
What to notice The data rule comes first. "Not sure yet" always sends you to start small and measure. Read each result's "why" and "not" lines: they are what you would say to the customer.

Why this check: to test the idea that the car pays off only when it is busy. Click one option, and you will see why it is right or wrong.

Next: the tree said "volume matters". But how high is high? Step 4 puts numbers on it, using Kavish's insurer.

Step 4: Do the sums: where is break-even?

Let's work it out together. All numbers are made up for the example. First, the inputs:

InputValueWhere it comes from
API price$2 per million tokensThe provider's price list
Pilot use800 million tokens a monthLast month's bill: $1,600 / $2 = 800
GPUs$4,000 a month for one setA quote for rented GPUs
People$2,000 a monthPart-time engineer and on-call time
One set can serve4,000 million tokens a monthA test run on that set

Now the sums:

  1. Pilot on the API: 800 x $2 = $1,600 a month. This matches the story.
  2. Self-hosting: $4,000 + $2,000 = $6,000 a month, used or not.
  3. Break-even: $6,000 / $2 = 3,000 million tokens a month.
  4. Why divide? Dollars a month divided by dollars per million tokens gives million tokens a month. It is the use at which the API bill reaches $6,000.
  5. The pilot uses 800 million. That is far below 3,000.

So the answer is: stay on the API for now.

Now try ten times the use. That is 8,000 million tokens.

  1. API: 8,000 x $2 = $16,000 a month.
  2. One set of GPUs serves only 4,000 million. GPUs come in whole sets, so you cannot rent 1.5. You need 2 sets: 2 x $4,000 = $8,000.
  3. Add $2,000 for people. Self-hosting is now $10,000 a month.
  4. Compare: $16,000 - $10,000 = $6,000.

So the answer is: at ten times the use, self-hosting saves about $6,000 a month on paper. Check quality and on-call first. Note that the $6,000 break-even used one set. The second set moves break-even up, but 8,000 is still above it.

What you are looking at: the two cases side by side. Each bar is dollars a month. The shortest bar wins.

Next: now try the same sums with your own numbers in the calculator.

Try it: the break-even calculator

Why this exercise: to see how fast the answer changes when you change one number.

The calculator follows one simple rule:

Its limits: it does not know about quality, traffic spikes, or how long setup takes. Treat it as a first filter, not a quote.

Do this:

  1. Leave the numbers as they are. This is the pilot at 800. The bar is short, and the text says the API is cheaper.
  2. Type 8,000 into the first box. The bar passes the dark line. A caution says you need 2 sets of GPUs, so the real fixed cost is $10,000, not the $6,000 in the first line.
  3. Set the first box back to 800. Click the word "Advanced" to open it. Set the last box there (millions of tokens the GPUs can serve) to 20,000. The caution now says about 4% use: "low, paying for idle hardware".
  4. Set the API price to 0.5. Break-even jumps from 3,000 to 12,000, so the bar fill shrinks. A cheaper taxi stays better for longer.

The bar rescales every time, so the dark line can stay in the same spot. Read the numbers and the length of the filled part.

Advanced

Break-even = fixed cost / price. Real life adds idle time, scaling steps and quality, so treat this as a first filter, not a quote.

What to notice The bar passes the dark line at break-even. The bar rescales, so watch the numbers too. A low API price pushes break-even far right, so the taxi stays better for longer. The cautions below the numbers are as important as the numbers.

Next: theory is done. Step 5 gives you two customer cases to practise on.

Step 5: Practise on two customer cases

Each case has a different clue that decides the answer. Click your answer first, and the reasons appear.

Scenario: the ticket sorter An online shop sorts 40,000 support emails a day into five labels. A provider's service is allowed. Traffic is steady, and a small tuned model matches a big one on their test questions. Finance wants a lower bill. Quick sum: 40,000 emails x 30 days x about 500 tokens each (a made-up size) = 600 million tokens a month. Compare that with the 3,000 break-even from Step 4.

Steady volume and a narrow job favour the car. But nothing is measured yet, so measure first. The hybrid answer: self-host the steady part, rent the rest. First step: count your monthly tokens, as in the sum above. About 600 million is well below 3,000, so measure before you buy anything. Prices here are made up, so use your own.

Scenario: the claim reader The insurer from the start must keep claim text in its own cloud account. It has no GPU experts. Volume is modest.

The data rule rules out a public API. No GPU skills rules out raw self-hosting. A managed service in their account fits both.

Next: a short list of wrong ideas that catch real teams, then a recap.

Mix-ups to avoid

Recap

In the field One team's own GPUs sat half idle. Day traffic did not fill them.

The lesson: busy GPUs are cheap. Idle GPUs are not.

Next: Kavish can now say "stay on the API for now, and here is the number that changes that". Before he can promise anything about a car, he must measure how fast it answers and how many people it serves at once. That is Chapter 33, Measuring latency and throughput.

Chapter quiz

Click one option for each question. Four questions. Score 70% or more to mark this chapter complete.

Review