What enterprise LLM inference really costs: open weights against the API

What the API invoice is actually made of

Hosted providers use the same two-part rate card, and the gap between the parts is the first thing to understand. OpenAI lists GPT-4o at $2.50 per million input tokens and $10 per million output tokens; Anthropic lists Claude Sonnet at $3 and $15. Output carries three to five times the input rate, so a workload that writes long answers spends most of its money on the output line.

Two discounts change that arithmetic. Cached input costs a fraction of the base rate: OpenAI halves it, and Anthropic charges a tenth for cache reads and 1.25 times to write one. Batch tiers take about half off again, for a turnaround measured in hours.

Long prompts are where the volume hides. A retrieval or agent workload re-sends the same instruction block, tool definitions and retrieved passages on every turn, so a fixed prefix of several thousand tokens is paid for again and again. Caching turns that per-turn charge into a per-change charge, and it is the largest lever on the API side.

The self-hosted side is a utilisation bet

Renting is the honest starting point: it puts both sides in the same units. AWS lists an eight-H100 p5.48xlarge instance near $98 per instance-hour in us-east-1, roughly $12.30 per card-hour, and Google Cloud sells H100 and A100 capacity by the GPU-hour too.

The rate does not fall when traffic does. That instance costs about $71,500 a month, idle or not, or about $8,900 per card, before storage, networking, egress and the engineers who keep the serving stack fed. A GPU at fifteen percent is the same invoice producing an eighth of the work.

Assume a 4-bit 70B model serves 2,000 output tokens per second on one H100 at realistic concurrency. That assumption, not the hourly rate, sets the answer. Fully loaded, one card yields about 5.3 billion output tokens a month and prices near $1.70 per million. At fifteen percent utilisation it yields about 790 million, and the same card prices above $11 per million.

What decides throughput before tuning starts

Memory comes first: a model that does not fit cannot be served at any speed. A 70B parameter model in 16-bit precision needs roughly 140GB for weights alone, which is two 80GB accelerators before a single token of cache is allocated.

Quantisation is how most teams avoid that. Storing weights in 4 bits cuts the same model to roughly 35 to 40GB, which fits one 80GB card with room to spare, at some cost in output quality. That trade deserves a measurement on your own evaluation set, not an assumption.

Context length competes for whatever memory is left. A long-context request on a 70B model can hold tens of gigabytes of key-value cache, so concurrency and context length are paid for from the same pool as the weights. Continuous batching, which admits new requests into a running batch rather than draining it, is what keeps the card occupied; the serving stack matters as much as the accelerator.

The costs that do not appear on the invoice

Procurement is a schedule risk before it is a cost. Cloud GPU quota is requested and approved rather than bought, accelerator supply moves in quarters, and a reserved commitment is cheap only if the workload arrives on the date the contract assumes.

Model lifecycle is the recurring one. Open weights ship new versions, and each upgrade means re-quantising, re-running evaluation, re-tuning prompts and clearing the regression set before it reaches production. That work does not disappear on the API side; it moves to the vendor's schedule.

The largest line is people. Somebody has to own serving capacity, autoscaling, upgrades, incident response and the token accounting that tells finance what the platform costs. A small platform team is a real annual cost, and at modest scale it is usually larger than the GPU bill it supervises.

When self-hosting wins on something other than price

Data that may not leave the estate is the strongest reason, and it does not depend on the arithmetic. Where a contract, a regulator or a client's own terms restrict where prompts and outputs may be processed, a hosted endpoint is not an option to be priced.

Residency and latency follow the same logic. A fixed internal workload with a hard latency budget and steady demand can be served on hardware you control, with no network hop and no shared capacity behind the endpoint. A genuinely steady workload is also the only one a vendor could discount far enough to compete, and vendors discount committed spend rather than your average.

Self-hosting trades a variable bill for a fixed one plus an operational obligation. That trade is good when the utilisation curve is flat and high, and bad when the workload is a pilot, a seasonal peak, or something the business will reshape next quarter.

The crossover, worked with the assumptions stated

State the assumptions or the comparison is a slogan. Take one H100 at roughly $12.30 per card-hour, a 4-bit 70B model, 2,000 output tokens per second, and a frontier API at $15 per million output tokens. Full utilisation gives about 5.3 billion output tokens a month at $1.70 per million, so break-even sits near eleven percent. Halve the throughput assumption and break-even moves to roughly twenty-three percent.

The number that matters is sustained, not peak. A team that hits eighty percent for two hours a day and idles the rest of the week is running a low-utilisation estate and paying for it. If the cards cannot stay above roughly a third busy across a full week, the API is cheaper even after the vendor's margin.

Compare like for like: a 4-bit 70B model is not a frontier API model, so the fair comparison is a mid-tier hosted rate, where break-even utilisation roughly doubles. Where quality is close enough for the task, self-hosting at high utilisation wins on cost.

The decision order is straightforward. Start on an API: it turns an infrastructure programme into a line item and produces the record of real token mix and utilisation the next decision needs. Move to open weights when that record shows high, steady demand no vendor will discount far enough, or when the data cannot leave the estate.