Nobody needs business class to fly from San Francisco to Los Angeles. The flight is barely long enough for the drink service, and whoever sits in the back walks off feeling just as fresh as whoever sits up front. A flight from San Francisco to Sydney is a different story. You’re on the plane for about 15 hours overnight, and the cabin you pick can decide whether you walk into Monday’s meeting ready to work or spend the day recovering. Most corporate travel policies account for this, setting the cabin by flight length and only allowing business class beyond a certain number of hours.
We bring up travel policies because enterprise spending on AI coding agents has nothing like them. Picture a CFO looking at the monthly AI bill. Every coding task flew business class, from a one-line fix to a bug in a critical payment service. Nobody chose that cabin on purpose. The team picked the model at the top of a public leaderboard because that’s what a leaderboard tells you to do, and nobody wants to be the person who picked the second-best model for the payment service. What the leaderboard doesn’t tell you is that most of those tasks never needed the top model. The top model also happens to be the most expensive one, so the company is paying several times more for work that should have flown economy. And nobody can say which tasks were the hop to Los Angeles and which were the flight to Sydney because nobody tested the models on the company’s own code.
At Everpure, we built an internal version of a coding benchmark using our private codebase and now use it to evaluate new models as they ship. We also moved our power users from using Claude exclusively to our on-prem GPU cluster (hardware we own and run ourselves), which routes routine work to the models we host and sends only the most demanding tasks to cloud frontier models (the largest and most expensive class of model, rented per use). That shift recouped the cost of our GPU investment in less than a quarter.
We’re releasing the public version of that benchmark so you can build your own. We rebuilt it on the open-source Zulip team chat application instead of our private code, and the tasks, harness configurations, grading, and raw results are all in the repository for you to copy and point at your own codebase. The specific tasks matter less than the method. Every result comes from a model running inside a harness, the software that drives the model by reading files, running commands, and deciding what to try next. We ran 21 model and harness combinations on 107 shared tasks, and each score is a single attempt per task, with no retries and no human guidance.
The delta between model results is exactly why the exercise is worth doing. Claude Opus 5 running in Claude Code harness resolved 94.4% of the tasks (fixed them correctly, as judged by a hidden automated test) at $3.68 per task. GPT-5.6-terra running in Codex resolved 85.0% at $0.54 per task, and the self-hosted open-weight model Gemma-4-31B (a model whose files you can download and run on your own hardware) running in the open source pi harness resolved 56.1% at $0.044 per task. That’s an 84X difference in price for a 38-point difference in resolve rate. No public leaderboard can tell you how well a given model will run on your specific tasks. The rest of this post walks through how we built the benchmark so you can run the same comparison on your code and write the cabin policy your AI bill is missing.
What we run at Everpure
Every task that reaches our coding agents gets routed, meaning the system picks which model handles it, either one running on our own cluster or a frontier model in the cloud. Here’s how we think about that choice: On most coding tasks, a cheaper model produces a very similar code change a frontier model would, so the frontier price buys nothing. On the hardest tasks, a cheaper model is unable to fix the bug. That’s where the frontier earns its place. Our on-prem cluster handles the first group. Frontier models get the second group, plus any overflow when the cluster is saturated, and the router still picks the cheapest frontier model that fits the task so overflow doesn’t turn into overspend. For how the data layer shapes inference cost once you host models yourself, see The Architecture Behind More Efficient AI Inference.
The hard part is knowing which group a task belongs to before you spend the money on it, and that’s what the benchmark is for. We built it from our own codebase’s history, so every task is a real change that a real engineer once made in our code. We run it against new models as they’re released, and the results tell us which kinds of tasks our hosted models handle and which ones still need a frontier model. The router uses that split to decide where each task goes. The payback in under a quarter is a measured result, but it reflects our mix of work, so your number may differ.
We’re not the only ones seeing these savings. AT&T reportedly cut its AI coding costs by roughly 56% for about a 2-point decline in quality by sending routine coding queries to open source models. NVIDIA’s open source router, NeMo Switchyard, showed a 74% cost reduction on one workload by starting with a lightweight model and escalating to Claude Opus when it detected trouble. All three results come from different workloads, and the industry is actively working on other solutions, too. Cost-efficient AI also depends on feeding models clean data; see Cost-Efficient, High-Precision Enterprise AI with Everpure Data Stream.
What the Zulip results show
Every model and harness combination was tested on the same 107 Zulip tasks. The resolve rate is the share of those tasks a combination resolved. Cost is the list-price API cost of one task. For the self-hosted models, we priced each task at the public per-token rate a hosting provider would charge to serve that model for you, so the number reflects what you’d pay to rent the model, not what it costs to own the GPUs, power them, and staff the cluster.
Every score is one attempt per task. The model gets no retries and no human feedback, and although it may take many steps on its own inside that attempt, nobody steers it, and it submits the fix once. With engineers guiding a model over several rounds, it would likely pass more tasks. But engineer time is what you value most, and you want it spent on your most complex work, so we interpret the one-attempt resolve rate as a measure of how much work a model can complete without consuming engineering resources.
Figure 1: The resolve rate for each model and harness combination.
Figure 2: The same combinations by cost against resolve rate for models running the pi harness only.
Price per task falls much faster than accuracy until you reach the smallest models. Compared with Claude Opus 5 in Claude Code, GPT-5.6-terra in Codex costs 85% less per task for a resolve rate 9.3 percentage points lower, and Gemma-4-31B costs 99% less for a resolve rate 38.3 points lower (comparisons throughout this post are our arithmetic on the measured values). The filled dots in Figure 2 are the combinations with no cheaper alternative that scores as well, and they mix cloud and self-hosted models. This is referred to as the “Pareto Frontier.”
The business question is how much of your work needs the additional 9.3 points of resolve rate that separate Claude Opus 5 from GPT-5.6-terra. The benchmark doesn’t say to run the cheap model everywhere. Gemma-4-31B resolves 56% of tasks, so it fails on the other 44%, and the value of the measurement is in learning which tasks fall on which side. That’s the information a routing policy needs.
There is a limit to what a single run can tell you. Our Zulip sample is small, and a difference of one or two tasks is noise, so the ranking treats combinations within 2 percentage points of each other as practically equal, and the cheaper one wins on the leaderboard. If your internal task set is also small, don’t read a one-point lead as definitive.
How the benchmark turns your code history into a test
Every mature codebase contains a record of problems that engineers have already solved. Each record is a tracked issue, a fix, and a test that proves the fix worked. Pull a few hundred of those records out, and you have an exam with a question bank, an answer key, and a grader, all written by your own engineers before anyone thought to test an AI with them. That exam is the benchmark, and every model has to take it to earn a place on your team.
For each question, the model receives the issue description and the code as it stood before the fix, and nothing else: no fix, no test, no commit message, no history. It edits the code and submits a change, and we grade that change by running the original engineer’s test against it. The result is pass or fail, with no LLM-as-a-judge deciding whether an answer looks right. Two safeguards keep the grading fair. Before a task counts, a fairness check (we call it the “oracle check”) confirms that the test fails on the unfixed code and passes on the real fix, run repeatedly to rule out tests that fail at random, and tasks that don’t flip cleanly are discarded rather than patched up. Grading then runs in a sealed container with no network access, and any edits a model makes to test files are reverted, so a model can’t pass by rewriting the test.
Why the benchmark has to run on your own code
Public benchmarks don’t measure the kind of coding your engineers do. Your team may work heavily in a language a public benchmark barely tests, and it follows conventions specific to how your organization writes code. The most widely cited benchmarks, such as SWE-bench, also draw from a small number of mostly Python projects that may look nothing like your codebase. On top of that, so many model developers now optimize for the public benchmarks that a high score has become a target in its own right. The industry term is “benchmaxxing,” which means tuning a model to score well on public benchmarks without a matching gain on real work. The only way to find out what a benchmaxxed model does on your work is to run it on your code.
Scale AI’s SWE-Bench Pro, published in September 2025, shows how far a public score can fall on different work. Most top models of that period scored over 70% on the widely used SWE-bench Verified, but on SWE-Bench Pro, a harder multi-language benchmark, the best models, GPT-5 and Claude Opus 4.1, each scored about 23%. On private commercial codebases, the same models fell again: GPT-5 from 23.1% to 14.9% and Claude Opus 4.1 from 22.7% to 17.8%. These are 2025 models, and the numbers don’t say why the scores fell, but the drop was steepest on private code, which is the kind of code your engineers write.
Your private codebase has two further advantages. Your code is never in a vendor’s training data, so a model that resolves your tasks is solving them and not remembering them. Running the benchmark internally also keeps the task set, scores, and evidence inside your environment, so the decision about which model touches which code rests on data you own. One caveat on privacy: The step where the model generates its change calls a live model API, so a cloud model does see the task, and only the grading step is sealed. If a codebase can’t be shown to a cloud model, run that step with on-prem models only.
How to use the open source benchmark
The repository ships the Zulip task set, runner, grading and isolation machinery, ranking math, and local dashboard that produced Figures 1 and 2. You can run the Zulip evaluation as it comes to compare the models and harnesses you’re considering or point the tool at your own repository, which is the version that informs routing and hosting decisions. Everything specific to Zulip lives in four places, so adapting it means changing four things:
- Mining rule. This defines which past commits count as tasks, using your convention for closing issues, directory layout, and naming of test files. It’s the highest-value change because a good rule produces fair tasks and a loose rule produces noise.
- Environment recipe. This builds a clean container that can run your tests at an old commit. Old commits have build quirks, so this step accounts for most of the roughly one to two weeks of one-time engineering effort.
- Test command. The only requirement is that a passing run reports success and a failing run reports failure.
- Model roster. This lists the models, harnesses, and providers you want to compare. The harness, isolation, scoring, ranking, and dashboard work unchanged.
On Zulip, roughly half of the 220 candidate commits became usable tasks. Your rate will depend on your repository. Every candidate, accepted or rejected, gets a written record of why, so a reviewer outside the team can audit the task set, which matters when the result drives a hardware purchase. Before the full run, have an engineer try one model on one task, which catches environment problems in minutes. The repository’s METHODOLOGY.md covers task validity, contamination, and the ranking rules in detail.
Adding a model is one entry in the roster file, and re-running the roster is a single command that runs overnight, so budget a few hours per new model release. The runs themselves cost money, and the bill grows with a bigger roster. At list price, one pass of Claude Opus 5 in Claude Code over the 107 Zulip tasks comes to about $394 per model and per run. Over time, teams narrow the roster to the models they judge best, add each new release as a candidate, and drop the ones that fall off.
Save business class for Sydney
A travel policy sets the cabin by flight length, and a public leaderboard is someone else’s travel policy, often written by the airline. It was meant for their routes, so following it means booking the cabin their trips needed rather than the one yours do. A benchmark built from your own history is how you write your own policy. It measures how long each of your flights is, meaning how hard each kind of task is, and which cabin gets you there in working shape, i.e., the cheapest model that still resolves it. With that in hand, you can run the hops to Los Angeles on hardware you own and save the frontier fare for the flights to Sydney. The Zulip numbers show the method because your code and your workloads will differ from ours. The time to start is the week you decide to host a model on your own hardware, not after the cloud bill forces the decision. The open source Zulip template is the place to begin. Once you host models yourself, the next question is scale: Production AI Doesn’t Just Need to Work. It Needs to Scale.
