When Does Owning Inference Pay?

An API is a good place to start. Owning the hardware begins to make sense when the workload, or the need for control, justifies the bill.

I have discussed running on-prem AI inference with a number of founders. Most consider it overkill to buy hardware for their own workloads. A few see it as a necessity. The disagreement becomes easier to understand once you separate the hope of a smaller bill from the need to control where the work happens.

For a new workload, I would usually start with an API. It lets you find out what the model can do before you take on the job of running it. Ownership becomes worth considering when you know what work you need done, which locally deployable model can do it, and what you gain by taking responsibility for the hardware.

The model question comes first. In the previous essay, models with the same benchmark score produced very different bills. A deployment comparison has to preserve the same discipline: measure the cost of work you can accept. Buying a machine does not make its cheaper tokens a substitute for a model that does the job better.

The cloud is a batching machine

A model API service gives you access to a serving operation whose job is to get as much useful work as possible from its hardware. Its customers supply the demand, and its software determines how efficiently that demand can be served. That combination is part of what you buy with an API.

Batching is one of the reasons this works. A serving system can process several requests together, sharing the cost of reading the model's weights across them. With enough concurrent work, the same hardware produces more tokens overall, although each person may have to wait longer for theirs.

SemiAnalysis's InferenceX measurements make the tradeoff concrete. Kimi K3 on B200 hardware delivers 1,007 total tokens per second per GPU when the target is 200 output tokens per second per user. Lower that target to 30 and throughput reaches 6,783. These are the best measured configurations at each speed, normalized per GPU, on InferenceX's AgentX workload, which replays Claude Code sessions with a median of 142k input and 444 output tokens a request. That is the shape of an agent loop, and the shape of the work I run.

The provider gets more work from the hardware it already pays for. How much speed you can give up depends on the work. An overnight processing job and an interactive conversation do not need the same thing.

The model matters as much as the speed. On the same GPU at 100 tokens a second per user, GLM-5.3, a 744B parameter model, serves 10,968 tokens a second, five times Kimi K3's 2,147. Kimi K3 has 2.8T parameters, and its measured configuration spans sixteen B200s, two servers rather than one. The figure carries both.

Measured 2 September 2026

Slow each user from 200 to 30 tokens a second and the same GPU serves 6.7 times the tokens

Caching gives the provider another way to avoid repeating work. If each request begins with the same instructions, tool definitions or document, the system can reuse the computation for that prefix. Anthropic's prompt-caching documentation describes how this reuse becomes a lower input price. In the replayed sessions, 95 to 98 percent of input tokens are cache hits, so cached input dominates the API bill.

You can use batching and caching when you run models yourself. What you cannot assume is that your workload will keep the machines busy enough to reproduce a provider's economics. Pooling demand from many customers gives an inference service more opportunities to fill its capacity. On your own hardware, the available work has to come from you.

Control over convenience

Cost is only part of the reason someone might choose to own the system. The nature of the business can make control worth paying for, even when an API would be cheaper.

The data cannot leave. A business may have data it will not send to an external model provider at all. Argonne National Laboratory described that requirement in its work on a local geospatial pipeline: control over model execution, no external API dependencies, and sensitive intelligence processed "without data leaving secure infrastructure".

Other buyers can get the control they need through a service agreement. In July, Norsk Helsenett, which runs the Norwegian health sector's network, announced a direct award to Anthropic for Claude Enterprise. Its stated reasons included a separate data processing agreement, control of where data is processed and better security mechanisms. For that buyer, more control was available through the contract.

Before buying hardware on privacy grounds, I would be precise about the boundary the data must stay within. Keeping it inside your building, inside a country and inside an agreed processing environment are different requirements. They need not lead to the same purchase.

The response cannot wait. Some work needs inference close to the device doing it. 1X, building the NEO humanoid, states the requirement directly: "Once models are trained, they need to run directly on the robot with minimal latency." For that kind of system, the acceptable delay is part of the design. An attractive average response time is little comfort if the slow responses arrive too late.

The model needs to stay available. Once a workflow has been tested against a model, a forced change means work. Providers can retire models, as Anthropic's deprecation notices document. Running open weights with a pinned runtime gives you control over when that configuration changes, whether the hardware is rented or owned. You take on maintenance, but you can plan the model change around your own system.

All of these are reasons to want more control. None makes a machine secure simply because you own it. A local network can restrict external connections, but access, patches, monitoring, backups and data transfers still need someone responsible for them. That responsibility is part of what you are buying.

Where ownership breaks even

Suppose the model can do the work and either deployment meets your requirements. Now the price comparison becomes useful. How busy would an owned machine have to be before it costs less than the service?

Start with the bill that arrives whether the machine is busy or not. An eight-B200 server starts at $400,328.50 at one reseller. Write it off over three years with no residual value and the hardware costs about $11,120 a month.

Electricity adds about $960. For this estimate I use Nvidia's 14.3 kW maximum draw for a DGX B200 throughout a 730-hour month, at the June US industrial electricity rate. Together, hardware and electricity come to roughly $12,100 a month for one server, or $2.07 per GPU hour.

This is a partial ownership cost. Colocation, staff, spares, networking and cooling overhead are left out, as is extra capacity to keep the service available through maintenance or failure. The calculation gives ownership the benefit of those omissions.

The amount of work that bill buys you depends on the model, the speed required, and how many servers the model needs.

GLM-5.3 runs on one. At 100 tokens a second per user, InferenceX measures 10,968 total tokens a second per B200. Across eight GPUs that is an assumed capacity of 7.6 billion tokens a day, and a token costs $0.05 at full use.

Kimi K3's 2.8T parameters do not fit in a server's 1,440 GB, so it takes two, $24,155 a month. At the same speed it serves 2,147 tokens a second per GPU. Across sixteen GPUs that is an assumed capacity of 3.0 billion tokens a day, and a token costs $0.27 at full use.

That is an extrapolation from the benchmark, not a measurement of the quoted servers. The benchmark's request shape is a median of 142k input and 444 output tokens, with most of the input served from cache. Your agent will land at a different rate. Before making a purchase, I would want the intended configuration measured on the actual work.

For the crossover, I give the API its strongest case and price every input token as a cache hit. For the benchmark's request shape, Z.ai's GLM-5.3 prices blend to $0.27 per million total tokens. One server meets that at 19% of assumed capacity, about 1.45 billion tokens a day. Moonshot's Kimi K3 prices blend to $0.35. Two servers meet that at 77%, about 2.3 billion tokens a day. The surveys below say almost nobody reaches that utilization.

Without caching the picture inverts, and that is the measure of what caching is worth. At the uncached prices, $1.40 per million input for GLM-5.3 and $3.00 for Kimi K3, the API bills come to $1.41 and $3.04 per million, and owning breaks even at 4% and 9%. This is the opposite bound. Agent loops reuse much of their prefix, and the AgentX sessions report 95 to 98 percent cache hits. The cache is the provider’s discount for work it does not have to redo, and it is most of the reason the API is hard to beat.

Renting the GPUs is a third option. At Lambda's listed $6.69 per B200 hour, renting breaks even with owning at about 31% for either model, because a rented GPU and an owned one serve the same tokens at the same throughput. The proviso is that every rented hour has to stay productive, and an always-available service with uneven traffic would have idle hours to account for.

9 September 2026

Owning GLM-5.3 breaks even at 19% of one server, Kimi K3 at 77% of two

Here, utilization means the work delivered over the month as a share of that assumed token capacity. The more of it you use, the more tokens that fixed bill spreads over. For GLM-5.3 on one server, the owned token is $1.05 per million at 5% and $0.10 at 50%. Those are scenarios to compare with your demand, not predictions of what your business will achieve.

Two things follow. The smaller model is the one worth owning, not because it is better, but because the same hardware serves five times the tokens with it and its maker prices the cache hit at five times what the owned token costs. The flagship is the one to rent by the token. Moonshot's cached price is $0.30, close to the $0.27 that serving Kimi K3 costs at full utilization on your own machines, before colocation, staff and spares.

The machine does not have to run at full capacity around the clock to cross the API price in this example. It does need enough work, at the required quality and speed, to pay for the capacity you committed to. If demand falls, the hardware still has to be paid for. If it outgrows the machine, you need more capacity or a longer queue.

That is why I would begin with the service and let the workload make the case for ownership. Measure what an accepted result costs, how much work arrives and how much of it can wait. If those measurements support the full cost of running the system, there is an economic case to examine. If the business requires local control regardless of price, the same measurements tell you what that control will cost.

Sources and measurement

Nothing here is a hardware benchmark run by Zero One Labs. Every input and calculation is in the dataset, generated by scripts/prepare-onprem-dataset.mjs.

• Throughput. SemiAnalysis InferenceX on B200, the AgentX agentic coding workload. Kimi K3: 13 configurations from 29 July to 28 August 2026, Dynamo vLLM in FP4, sixteen physical chips. GLM-5.3: 5 configurations on 2 September, SGLang in FP4, eight chips, in a sweep that stopped at 16 concurrent clients, so its 30, 50 and 75 rungs are unmeasured. The batching figure takes the best configuration at each per-user speed. The cost calculation takes the 100 tokens a second point, the slowest both models reach.
• Workload. AgentX v1.0, 393 replayed Claude Code sessions, median 142k input and 444 output tokens a request. Input is 98% cache hits in the full set and 95% in the 256k variant. Which variant each run replayed is not printed on the run page.
• Tokens. Blended prices and capacity count input and output together, cache hits included, on both sides of every comparison.
• Servers. One reseller’s listed starting price for an eight-GPU HGX B200 system, retrieved 8 September. 36 months of depreciation, no residual value. Kimi K3 is priced at two of them, the smallest footprint its measured configuration fits.
• Power. Nvidia's stated maximum draw for a DGX B200, applied for all 730 hours of the month, at the EIA's June 2026 US industrial average.
• API. Z.ai’s GLM-5.3 and Moonshot’s Kimi K3 list prices, retrieved 9 September. The crossover prices every input token as a cache hit; the prose also gives the fully uncached bound.
• Rental. Lambda's saved hourly rate at the same assumed throughput, every rented hour productive. Startup time and idle or reserve capacity would raise it.
• Images. The batching and crossover figures are under /research/on-prem/, with SVG originals and a manifest recording the dataset hash.