The Cheapest Model Is Not the Cheapest Model

We built an evaluation harness around a production agent, tested models on the workload it performs, and cut inference cost by 86% while preserving reliability.

Inside the commission reconciliation platform we have built for a client, call them client X here, is an agentic feature that lets the user cross-verify their data, check corrections, and do whatever else is useful with it. One of the key problems we faced in delivering this service was that the cost of the model powering the agent quickly started to eat into the margins.

This is why it was difficult

The model we had chosen was a specific choice, and it gave the system the right capabilities at the right price at the time. The agentic feature was added in February, on Claude. By summer it ran on Claude Sonnet 5 with adaptive thinking and a one-hour prompt cache, and that is the configuration measured below. At the time it was the right choice, balancing cost, speed, data retention and developer velocity.

Sonnet was also the easiest choice, because we did not see the value in going for a less tested open model. Keep in mind that this was February. The GLM-5 report came out on 17 February, claiming state of the art on the major open benchmarks in its own abstract, and the open models that followed it through the spring were still to be proven on anything like our workload. Since then, the market of open models has grown greatly in count and quality.

So the agent for client X was ripe for a round of model choice optimization.

This is what we changed

The first requirement with this kind of system, which handles large amounts of sensitive data and needs a higher level of environmental control, is to consider which model API services meet the data retention requirements while also meeting the quality and cost targets that are the foundation of the work.

Through my experience with a wide range of providers, I do have a selected few. For the client X system I chose Baseten, which provides accounts with zero data retention out of the box. In addition to meeting the privacy requirement, Baseten brings up the latest models quickly and has run an inference service of the highest quality for years.

Beyond the provider comes the evaluation harness and the workload tasks to run each configuration against, to find the suitable choice. In the client X system we already had seven months of chat history in the database, about 400 user messages across 90 conversations, which we had access to and from which we could build the evaluations. This let us simulate the specific use cases accurately and find the exact tradeoff between cost, quality and intelligence.

The tests we created were mainly about the model using a SQL tool against a read replica of the customer database. The agent writes SQL, receives the result into a sandbox, and runs commands there to read and use the data. Other tasks were to process a pasted or uploaded file into the agent's sandbox, from where it uses the same commands to compare the file against database data, to check for differences or to help the user understand what they have.

The beauty of this kind of agentic system is that its outputs can be validated and checked. What this fundamentally means is that when testing a wide range of models, I can programmatically verify whether each model's output is correct. With that feedback loop in place, the model evaluations scale with much less work.

I designed the evaluations to work through the problems users had already used the system for, and to verify the agent's output against the data in the customer's database. The harness is 22 cases with deterministic gates: a figure resolved from the database at run time that must appear in the answer, a tool that must have been called first, a table that must reach the panel with the right row count, a write that must be proposed and never executed, Norwegian orthography that must survive. Six of the cases attach a file or a paste and run in the sandbox. Every case runs in-process against the same tools and the same prompt, and the eval runner records the tokens the provider reported and what they cost at list price.

Keep in mind that there are a lot of different model configurations. From the effort level (an approximation of the size of the reasoning budget), to which inference provider (and their model deployment configuration) is used.

In our comparison below we look at the previous model used for client X system, with the new model configuration, and comparing the costs before and after on the same cases.

In its production configuration: Sonnet 5 with adaptive thinking, and GLM-5.3-Flash on Baseten at max effort. GLM-5.3-Flash passed 62 of 66 runs across three first repetitions, succeeding once environment error was fixed.

Measured 9 September 2026

Cheaper on all 22 cases, by 4.6 to 12 times a turn

Cost per turn is where the two separate. The shared prefix, the system prompt and the tool definitions, is about 17,000 tokens on Sonnet and 11,000 on GLM. So most of the bill is the input tokens that get cached prefix. With the prefix served from cache (which is what an iterative agent loop does), the turns after the initial message is billed at, $0.012 per turn on Sonnet 5 and $0.0016 on GLM-5.3-Flash. That is a saving of 86% per turn, on every one of the 22 cases, GLM is cheaper by between 4.6 and 12.3 times.

The comparison harness starts a fresh conversation when running each case, which means that we loose out on the cache read for the first step. Priced that way the run cost Sonnet $1.48 against GLM $0.11, a saving of 98%. Running a cold (not cached) test is the honest bill for a workload of mostly one-turn conversations, and the 86% figure is the honest bill for a session that continues (warm, cache hit).

Measured 9 September 2026

Faster on 15 of 22 cases, and seven times slower on one

Speed went the same way for most of the set. GLM-5.3-Flash was faster in 15 of the 22 cases, with a median of 5.8 seconds versus 7.5, and the difference on direct lookups was about a second and a half. GLM was slower on seven cases, and badly so on one: a formula question that Sonnet answered in 3.6 seconds took GLM 25 seconds at max effort, almost all of it spent reasoning. That is one downside of running GLM at max effort.

A practical solution to that would be to configure a timeout on each API call so that, if the model exceeds X seconds, a new API call immediately fires at a lower reasoning level. Given GLM’s lower cost, this is a pragmatic way to eliminate pathological over-reasoning. The obvious qualification is the task itself: if it genuinely requires the additional reasoning time, then permit it.

We have so far discussed only two models, Sonnet and GLM 5.3 Flash. This is after having iterated the evaluations over the models and configurations available. I converged on GLM-5.3-Flash as the suitable option for this workload.

Measured 9 September 2026

The larger open model costs 1.6 times Sonnet 5 a turn and passes fewer gates

The effort setting is another axis the evaluation measures. On direct lookups, GLM-5.3-Flash at low effort responds in a median of 2.2 seconds versus 5.0 at max, spends 14 reasoning tokens per turn versus 711, and costs $0.0012 per turn versus $0.0016. High effort sits between the two.

GLM-5.3 at max effort (the larger sibling of GLM-5.3-Flash, fallback behind Flash in production) passed 39 of 44 runs, took a median of 12.6 seconds, and exceeded the two-minute budget on both runs of the largest sandbox case. At $0.019 per turn, it cost 1.6× as much as Sonnet 5 on a cached prefix and 12× as much as Flash.

This exposes a distinction that headline API prices hide. Sonnet 5 costs $2 per million input tokens and $10 per million output tokens, compared with $1.40 and $4.40 for GLM-5.3. On paper, GLM is cheaper. But on the largest sandbox case, running the models side by side reverses that relationship: GLM-5.3 costs 1.6× more per turn than Sonnet 5.

An evaluation harness is what lets us measure the practical cost of a model rather than infer it from the price of its tokens.

This is why that intervention worked

This cost reduction is possible because of the accelerating capabilities of open models, and because the bounded data on the existing user tasks allows a programmatic feedback loop that makes the model's output verifiable. That is different from a qualitative task, like the creation of a design or a website, which is much harder to quantify or validate because its output is judged by a person's opinion of its quality.

I believe that spending time understanding the evaluation harness as fully as possible, and considering creative ways to build feedback loops that validate a model's performance on a system, is very high value work. It is the precedent that allows every downstream evaluation of a model configuration.

When I have an evaluation that can be programmatically verified, I can scale the number of samples. More samples give higher resolution, because the distance between what was observed and what is inferred from it shrinks. Through such feedback loops I reach the scale at which I can make informed judgements about which model achieves the best outcome. On this set, three repetitions were enough to find that one failure was a provider defect that recurs on every run, and that another was a reasoning budget that occasionally overruns.

This is the measurable result

Measuring the v1 system against the optimized v2, the cost per simulated user task fell by 86% on a continuing session and by 98% on a fresh one, while the answers stayed accurate and reliable. Through the harness, the savings were low-hanging fruit, readily available to collect.

The greater picture is that open models are now cheap, fast and reliable enough that most of the workloads we run could be moved to them, for a more economically effective system overall.

One of the greater potentials here is the increase in margin, which can be used in two ways: retained, or reinvested in a more expensive open model. Since we have cut the cost of using a closed model, that capital is freed and could be spent on a more intelligent model at the same cost. That is an appealing option if users start doing more complex tasks and the cheaper model falls short. The margin can then be used to step up the model on those tasks, to increase the probability of success.

I enjoy this kind of discovery because it reveals that potential of a system needs to be modelled and discovered. The reward is greater operational efficiency.

Sources and measurement

Every number above comes from the platform's own eval harness, run on 9 September 2026 against a branch of the customer database, and the per-case results are in the dataset. The answers, the tool arguments and the progress labels are not in it, because they can carry figures from the customer's data. Nothing in the dataset names the client, a user or an agreement.

Prices are list prices read from the providers on the same day: Anthropic's pricing page for Sonnet 5, at $2 per million input tokens, $4 for a one-hour cache write, $0.20 for a cache read and $10 per million output, and Baseten's models endpoint for GLM-5.3-Flash, at $0.15 per million input, $0.03 cached and $0.50 output. Reasoning tokens are billed as output on both. The cost in the figures is the provider's reported usage at those prices, with the prefix on Sonnet priced as a cache read. The cold figures price the cache write the harness actually paid.

Sonnet 5 ran once over the 22 cases. The remaining cells ran two or three times each, and a case's figure is the median over its runs. Wall time is measured from the question to the last token and includes tool calls, database queries and the sandbox.

The figures are also available as images under /research/model-selection/, with SVG originals and a manifest recording the dataset hash.

This is the kind of work we do at Kylace: evaluate and improve AI workloads against the requirements that matter for the system. We test changes across models and the wider system on the actual work, then turn the results into a reusable evaluation and a recommendation your team can act on.