The hidden mismatch: Evaluating coding agents on the work developers actually do

In brief

We’ve all heard it before: your AI system – whether an agent, harness, or routing layer – is only as good as the evaluation behind it.

We’ve also heard: public benchmarks are poor proxies for real developer traffic. In terms of what they ask for, how they phrase it, task difficulty, and how trajectories unfold over the session.

We saw this discrepancy up close when we surveyed our internal developer traffic. The results challenged our assumptions about agentic coding tasks, and we used these lessons to build a robust benchmark that preserves quality on real-world coding tasks in our intelligent gateway. This post covers that journey.

The real distribution of developer work

Luckily, to understand the gap between public benchmarks and real usage, we have the benefit of real live data from our internal team of developers. To make sense of this traffic, we developed an internal coding task classifier and ran seven weeks of real developer traffic through it.

What we saw when we looked at the resulting task distribution completely challenged our intuition about agentic coding habits:

FIGURE 1. Top-level categories were derived from the data, and subcategories were derived inside each category. Total of 15 categories, 84 subcategories, and 4,211 turns.

If you look at the tasks measured by a standard benchmark like SWE-bench, it would seem that developers use agents primarily for pure code generation and fixing bugs from a clean slate. But our classifier revealed a totally different reality.

Lessons from seven weeks of real-world developer traffic

Developers use agents as thought partners, not just as code generators. In our traffic, only about 11% of real developer turns ask the agent to write code. The vast majority of the time, developers are asking clarifying questions, pasting context, doing analysis, or requesting reviews. The single most common message we see is a clarifying question.

Interactive workflows vs. single-turn tasks. The data shows that developer-agent interactions are highly conversational. Developers aren’t just dropping a 1,000-word GitHub issue into a chat and walking away. They are typing short, 80-character messages mid-session. They are debugging step-by-step, pasting console logs, and iterating on the agent’s output.

Where real-world developer traffic and public benchmarks diverge

Task type

We ran several public benchmarks (including SWE-bench, HumanEval, and Spider) through this same classifier to build out our comparison. The mismatch with our internal data was glaring. Almost all of these public benchmarks focus on writing code from scratch or fixing a bug to pass a test.

FIGURE 2. The same categories applied to real usage and to 23 benchmarks. Number in cell reflects % of a dataset’s message. Confirmations, redirections, git commands and pasted context make up about 40% of real user turns, yet appear in no coding benchmark.Despite the fact that confirmations, redirections, git commands, and pasted context make up about 40% of real developer turns, these interactive properties and context-sharing behaviors appear in almost no public coding benchmarks.

Task difficulty

Beyond not covering real types of coding work, public benchmarks also fail to accurately reflect task difficulty. Standard evaluation sets often test deep, architectural knowledge of a codebase’s internals. But in the real world? A developer is much more likely to ask a quick, practical question like, “Why do I not see my changes in git status?”

Because benchmarks over-index on these highly complex, multi-file architectural issues, they give us a skewed sense of what models actually need to do and can therefore make smaller or cheaper models look artificially incapable of handling everyday traffic. You can see this in the comparison below between the median turn count across tasks from real developer sessions versus a frontier model tested on our benchmark:

FIGURE 3. Median turn count across task categories, as compared between real developer sessions and data generated from running Claude Opus 5.5 on our benchmark.

Task context

We found another major difference between real-world and public benchmark traffic: context. A typical benchmark task is presented as a well-written,1,000-character GitHub issue on a clean code checkout.

But real developer messages are messy. Averaging around 80 characters, they arrive right in the middle of an active session, stacked on top of an extensive history of conversation, tool outputs, and half-finished file changes. Over 60% of our internal sessions contain more than one distinct task. If you evaluate an agent solely on how it handles a single, well-scoped prompt from a pristine start, you are ignoring the reality of mid-session context switching.

Building a real-world benchmark

Defining the goal

Proving the discrepancy between public benchmarks and real-world developer traffic was the easy part. Now, we had to build a new benchmark that would resolve the gap.

The ideal evaluation would simply use our own developers’ real workflows. But building a natural task distribution creates an immediate tug-of-war between three competing properties: drawn from real work, context-rich, and capable of being graded automatically.

Real developer sessions meet the first two criteria, but are almost impossible to grade automatically. There is no pre-written test suite or rubric for a developer asking, “Why is my database connection dropping on line 42?” The intent lives entirely in the developer’s head, making it hard to measure if an agent succeeded without human review. On the other hand, public benchmarks are perfectly runnable and easy to grade automatically, but as we saw earlier, they lack relevance to real-world context.

Our goal was to collapse the tradeoff: combine the automated gradability of public benchmark tasks with the messy reality of actual traffic. We built a new benchmark, guided by four principles:

1. Let real traffic set the quotas. Instead of randomly sampling public tasks, we used the insights from our internal classifier to set quotas. If 40% of our real traffic consisted of clarifying questions and codebase analysis, then 40% of our benchmark needed to test exactly that.

2. Mix public tasks with internal code. Public tasks can’t cover your internal libraries, custom environments, or company conventions. To make the benchmark realistic, we mined our own internal repositories for tasks that had already been solved and merged. We verified that these internal tasks could be graded by our own test suites, and then added them to the pool alongside public tasks. That process looks something like this:

FIGURE 4. Mining GitHub and public benchmark tasks to create a comprehensive benchmark that covers expected real-world coding habits.

Synthesizing from traces involves analyzing real user interaction logs and session recordings to recreate realistic developer tasks and multi-turn scenarios.

3. Write like a developer. A 1,000-word, perfectly formatted bug report doesn’t test an agent’s ability to understand the short, 80-character, often ambiguous messages that our developers actually type. We changed the wrapper, but kept the underlying code environment and automated tests identical.

4. Start in the middle. Finally, because developers rarely ask for code from a clean start, an evaluation must eventually account for history. We built internal replay infrastructure that allows us to restore a real developer session at any point in its trajectory, complete with the accumulated context and modified files. By simulating the developer’s next move from the middle of a messy session, we get the truest measure of how an agent will actually perform in the wild.

By carefully selecting tasks that matched our true category mix, rewriting the prompts to sound like real developers, and blending in our own codebase, we created a 613-task benchmark that mirrors reality.

Then we put our benchmark to the ultimate test: measuring the quality of traffic flowing through our intelligent Gateway.

Does it hold up? Stress-testing our real-world coding benchmark

To test how well our benchmark captures the natural distribution of real-world coding tasks compared to public benchmarks, we wanted to see how a popular frontier model behaves on real user sessions versus each of these two benchmarks. For that test, we turned to the frontier-powered router in our intelligent Gateway, which sits between our developers and foundation models and has been trained for agentic coding traffic. Every time a developer sends a message, the Gateway’s router scores the task difficulty to make a split-second decision: Does this request require our most expensive, highly capable premium model, or can a cheaper, faster model handle it just fine?

This router can be tuned to route an expected percentage of traffic to the premium model, allowing us to trade off cost and quality. We reasoned that we’d know if we succeeded in building a truly representative benchmark if the shape of how the Gateway routes requests is consistent between real user sessions and our benchmark.

We measure that shape by looking at the premium routing threshold: the percentage of tasks the Gateway sends to the premium model. We set that target number at 25% and compared how the routing percentage correlates across live developer traffic and different benchmarks:

Test #1: Measuring our Gateway’s live traffic against a public benchmark (green line). Deployed on real incoming traffic, the Gateway misjudged the traffic and ended up sending nearly 50% of the requests to the premium model, overstepping its target budget by 23 points.

Test #2: Measuring our Gateway’s live traffic with our benchmark. Showing a striking correlation with our natural coding data, the Gateway landed just 0.5 points away from its target budget. This demonstrates that we have not only matched the task distribution of our real user sessions, but that the Gateway also successfully routed hard tasks to the premium model at the same rate in both cases.

FIGURE 5. The Gateway’s router threshold was set on its own data to send a target share of tasks to the premium model (x-axis), then applied unchanged to 1,905 real developer messages from our Gateway (y-axis).

Because the router was finally evaluated on a distribution that matched actual coding habits, it didn’t just save money; it proved that it could correctly identify when to rely on a cheaper model for a simple git question, and when to bring in the heavy hitter for a complex feature request.

What this means for your team

Ultimately, whether you are building a router, tuning a prompt, or choosing a model for your organization, your results will only ever be as accurate as your evaluation. If you want to know how a coding agent will perform in the wild, that’s where you have to measure it.

Here are the core takeaways for any team evaluating coding agents:

• Measure your traffic before you choose a benchmark. If 90% of your team’s requests are codebase questions and bug triage, testing your agents on a benchmark made entirely of code-generation tasks will lead you astray.
• Match the mix, the difficulty, and the language. Rewriting prompts or re-weighting tasks alone isn’t enough to transfer to real life. You have to combine them, and ideally, introduce tasks mined from your own private codebases.
• Price sessions, not single tasks. Because developers work conversationally, model switching inside a session costs real money and context tokens. Only a session-level evaluation can see that true cost.
• Test decisions, not just scores. A benchmark score doesn’t matter in a vacuum. An evaluation is only trustworthy when a model choice, a routing policy, or a budget threshold set on it actually holds up on real, live traffic