Dev48
Language
  • About
  • Services
  • Industries
  • Technologies
  • Articles
  • Contacts
Book a call
    Home/Articles/Making open models work for long horizon agentic workloads
Dev48
info@dev48.org+375 29 320-40-96WhatsApp@isky_by

© 2026 · All rights reserved.

Making open models work for long-horizon agentic workloads

Source: Together AI•November 16, 2026 at 09:43 PM

Making open models work for long-horizon agentic workloads

Every agent turn resends the full session. Together AI and Ollama break down how to protect the prefix cache, from the harness to the GPU.

Coding agents have an odd traffic pattern. Every time the agent takes a step, the harness sends the model the entire conversation so far: the system prompt, every tool definition, your request, every file the agent has read, and every command it has run. What comes back is usually a tool call a few dozen tokens long. Then it happens again, with everything from last time plus a little more.

This large input sequence length (ISL) and small output sequence length (OSL) is characteristic of most agentic workloads. If the GPU had to work through that whole transcript from scratch on every turn, long sessions would get slower and more expensive with every step. Prefix caching is what saves most of that work. The problem is that if you’re not careful it's also very easy to break the prefix cache.

We come at this problem from opposite ends. Ollama works on harness integrations which means deciding what goes into each request. Together AI works on the inference engine which means deciding what happens when that request arrives. We kept finding that much of our work on each side is about protecting the cache from things the other side can do to it, so we wanted to write it all down in one place to help agent and infra builders alike!

What an agent turn looks like to a GPU

Every request goes through two phases. The first is prefill: the engine reads the whole prompt in one big parallel pass and produces the first token. GPUs are good at this because it's a lot of math done all at once which maximizes arithmetic intensity. Prefill determines how long you wait before any new tokens appear. The second is decode: the model writes the rest of the answer one token at a time, and each new token requires reading back and attending through everything before it. Decode is limited by memory bandwidth more than by compute, and it determines how fast the text streams out.

A normal chat response spends time in both phases while an agent turn is mostly prefill, because the input is the whole session and the output is usually a short tool call. Reasoning models add thinking tokens on top of that, sometimes a lot of them, but as a session grows into the tens of thousands of tokens the input still dominates. Each turn the input gets longer, and the output stays roughly the same size — a few tool calls.

So for agents, "how fast is this model?" mostly means "how fast can it get through a long historical account of pervious turns?" The way to process a task with lots of previous turns quickly is to just cache it and read it back in rather then recomputing.

Cache is king

The easiest way to picture the KV cache(or even the prefix cache, for that matter) is reading. Imagine that to read each new word on a page, you had to start again from the top. The further down the page you got, the slower you'd go, because you'd keep having to reread everything you'd already read. That's a model with no cache. The cache solves this within a single request: the engine keeps the work it did on earlier tokens(stored as intermediate key, value pairs) and computes only the new one.

Prefix caching does the same thing across requests. Turn 7 of an agent session is turn 6 plus a bit more, so if the engine still holds turn 6's work, it only has to process the new part.

The arithmetic is simple. If each turn adds the same amount, ten turns without caching process 1 + 2 + … + 10 = 55 units of prompt. With caching, they process 10. At a hundred turns, which isn't unusual for a coding task, it's 5,050 versus 100. Real turns vary in size, so treat these as idealized numbers. A cached session isn't completely free either, since every new token still has to attend to the cached ones. But the pattern holds: without caching, the work you redo grows with the square of the session length.

The catch is that a prefix cache matches from the beginning of the prompt up to the first token that's different, and stops there. Everything after that first difference has to be processed again.

Adding to the end of the prompt costs nothing extra. Changing anything earlier, even a single character, means re-processing everything after it. If the change is near the top, you've effectively thrown away the whole session and now must rebuilt the cache. Almost everything else in this post follows from that one rule.

The harness decides where that first change lands. The engine decides whether the cache is still there when the next request shows up. We'll start with the harness.

The harnesses and APIs

On the Ollama side, we shipped the ability to connect your favorite harness through the ollama launch command. So you can connect harnesses like Claude Code, Codex, Pi (and more) with open models like DeepSeek-V4.1-Flash and GLM-5.3 and maintain your existing workflows.

Connecting a harness to an API might seem easy, but there are many layers of translation and varying expectations between a harness and a model. This often is the cause of cache breakages (incurring time and monetary costs), unexplainable agent behavior, and the harness outright erroring.

Loading tools late

Harnesses carry quite a few tools now. If every definition goes into the prompt up front, you spend context on tools the task will never use, and the model gets worse at picking the right one. Newer harnesses have added a feature for tool search to alleviate this, where the tool can be loaded and used by the agent itself after searching for it.

Let’s say you have a personal assistant as an agent which has this capability. If you ask the agent to export the events you've starred to a calendar, and it searches for "calendar," loads the save_calendar definition, and calls it.

For the cache, the question is where that definition goes.

The obvious approach is to add the definition to the tool list. That works, but the tool list sits near the top of the prompt, before the conversation. So if the prefix changes everything after it gets processed again. On a long session, that can be hundreds of thousands of tokens. The better approach is to put the definition at the end, where new content goes anyway.

Except not every model supports that. Some chat templates don't expect a tool definition halfway through a conversation, and the call errors out. So whether a harness can load tools late without paying for it depends on the model. Users shouldn't have to know this, but anyone building a harness does.

Things at the top that change

The easiest way to wreck the cache is to put something in the system prompt that changes from turn to turn, like a current timestamp, the git status, or a working directory the agent keeps moving through. If the harness re-renders the system prompt with fresh values on every turn, every turn is a full re-read.

Things that are fixed for a session but differ between sessions, like the user's name or the date the session started, don't hurt that session. They do stop new sessions from reusing a setup that's already cached, which we'll get to on the engine side. Either way, the fix is the same: put that information in the task message, once, and keep the system prompt identical everywhere.

Reasoning effort

We hit this one on the Ollama side. In GLM-5.3-Flash, reasoning effort isn't a separate setting the engine handles on the side. The chat template writes it into the very first system block, ahead of the tools and the conversation.

Let’s say you're deep into a long task and it isn't going well. You raise effort from low to high. That's a reasonable thing to do, but the next request now differs from the last one almost at the first token. Which means the whole session gets processed again. On a long run, that can be hundreds of thousands of tokens spent on changing one setting.

Our advice is to pick an effort level at the start and leave it there. If you really need more thinking, it's often cheaper to start a fresh session at the higher setting than to continue the old one, and you get a clean context out of it too. You could also compact before changing the reasoning value to decrease the amount of prompt tokens.

Big tool results

A tool’s output becomes part of the next request, and of every request after it. If the agent reads a 3,000-line file when it needed forty lines, you don't pay for that once; you pay for it on every remaining turn (even if it is cached you lose out on the context window). To combat this, harnesses often cut tool output down. That's the right call, but how you cut tool output matters.

Here's a failure that's easy to build by accident. The task is to fix a timezone bug in a calendar export. The agent reads calendar.py, and the tool returns the first 100 lines of a 640-line file without saying it stopped. The function that matters, convert_timezone, is at line 420. The model has no idea it's missing anything.

The fix is a single line in the output, telling the model it got lines 1 to 100 of 640 and that it can search the file or request another range. With that, the model searches for "timezone," finds line 420, and reads lines 410 to 450. Cutting output saves money. Cutting it without saying so gets you a model confidently editing code it never saw.

Subagents

Open models like Kimi K3, GLM-5.3, and GLM-5.3-Flash are now pretty good at splitting a big task and handing pieces to subagents. How the harness starts each subagent decides how much of the parent's cache it can reuse.

One option is for the subagent to inherit everything: the same system prompt, tools, and conversation, plus its assignment. Side chats work this way. They branch off the parent session, so the whole prefix is already cached. The other option is to send a brief: the same system prompt and tools, then only the files and instructions the subagent needs. In that case the cache covers the setup and nothing more.

Inheriting sounds strictly better, but it isn't. A cached prefix is cheap to process, not free. Every token the subagent generates still has to read through it, and it occupies GPU memory the whole time. A reviewer that needs two files and a checklist probably doesn't need three hundred turns of history. And if you give a subagent its own system prompt, it shares nothing with the parent at all, not even the setup.

The engine side

You can do everything above perfectly and still not benefit. The cache has to still exist, and the next request has to reach the machine that holds it.

At Together AI, we think of an inference engine as a machine for keeping GPUs busy. A single GPU costs around $50k to rent per year, so ideally it should never sit idle, and at any moment it's serving hundreds of users who are each at a different point in their request. Agent traffic pushes on those constraints in a few specific ways.

Routing requests for optimal cache hits

A session's cache lives on a particular node in a particular data center. If the next turn in a conversation lands anywhere else, that node has to process all earlier turns from scratch, and now two nodes are storing the same prefix. In order to route correctly we First need to pick a data center that runs the model, has capacity, and is where the session has been running. Then we need to pick a node inside it that has the previous turns cache.

There are two main ways to pick the node. The simple one is to use a key: the client sends something like prompt_cache_key with every request and then ideally we route every request with the same key to the same node. The other way is to hash the prefix and send the request to whichever node has the longest matching prefix cached, even if another session put it there.

That second option is the other reason to keep the system prompt identical. If every session of a harness starts with exactly the same system prompt and tool definitions, a brand-new session might be able to find that setup already cached. If you put unique identifiers in there then every session is guaranteed to start cold.

It also isn't as simple as always sending the request to the warm node. Sometimes the warm node is overloaded with traffic, and then the choice is between waiting in line there or taking an idle node and paying to process the prefix again. In this load-aware routing paradigm sometimes the cache miss might be the better deal. So even a perfectly stable prompt doesn't guarantee a hit if the infernce engine on the backend is not implemented correctly.

Keeping the cache alive between turns

Routing only helps if there's still a cache to route to. An agent will often make a tool call and then wait a couple of minutes while tests run. Keeping that session's cache in GPU memory the whole time takes space active users need, and throwing it away means re-reading everything when the agent comes back from its tool calls. So we can also offload the cache to CPU memory while the session is idle and restore it for the next turn.

Memory is the constraint behind most of this. We can't know ahead of time whether a response will be 30 tokens or 30,000, so instead of reserving room for the worst case max context length scenarios, we hand out cache memory in small blocks as each request needs them (PagedAttention), the way an operating system handles RAM. And we rebuild the running batch on every pass (continuous batching), so when an agent fires off five subagents at once, each new request takes a slot the moment one frees up instead of waiting for the slowest one in a batch to finish.

The upshot for anyone building a harness is that headroom for cache memory, not compute, is usually what limits how many requests a GPU can serve at once. Every oversized tool result, and every subagent carrying irrelevant history takes memory that another session's cache could have used.

Agents also push the prefill/decode split to its extreme, with enormous prompts and tiny answers. To better optimize this we run the two phases on separate clusters. One is tuned for compute and does prefill; the other is tuned for memory bandwidth and does decode for hundreds of streams at once, with the cache shipped between them over RDMA. That way, a wave of long agent prompts doesn't slow token streaming for everyone else.

Putting it all together: One session, both sides

Here's what all of this looks like in a single session.

On the harness side, the agent finds the calendar tool and adds it at the end of the conversation, so nothing above it changes. It reads calendar.py, sees that it got only lines 1 to 100 of 640, searches for "timezone," and reads the right section. Effort stays where it started. A reviewer subagent inherits the session to check agenda.ics against sessions.json.

On the engine side, every turn goes back to the node that has the session cached. While the tests run, the cache waits in CPU memory. The reviewer's request slots into the running batch, and each turn processes only what's new.

None of that is visible to the person using the agent; they just see it stay fast at turn ninety. But it only works if both sides do their part. A harness can keep the prompt perfectly stable and still lose the cache if the engine sends turn 7 to the wrong machine. An engine can route perfectly and still lose it if the harness changes the first line every turn.

Parth writes more about building agents at parthsareen.com, and Ollama posts updates at ollama.com/blog. You can also read more about and run the open models mentioned here on https://www.together.ai/blog.

← All articles

What this article says

More in AI & Machine Learning

All →
At Holiday Extras, dots get to work before anyone asks
OpenAI

At Holiday Extras, dots get to work before anyone asks

deepset | Digitale Souveränität bedeutet heute auch wirtschaftliche Unabhängigkeit: deepset macht die Kosten von KI-Agenten transparent und steuerbar
deepset

deepset | Digitale Souveränität bedeutet heute auch wirtschaftliche Unabhängigkeit: deepset macht die Kosten von KI-Agenten transparent und steuerbar

Kotoba is joining ElevenLabs
ElevenLabs

Kotoba is joining ElevenLabs

How the World’s Most Advanced AI Infrastructure Supply Chain Invents the Future
NVIDIA

How the World’s Most Advanced AI Infrastructure Supply Chain Invents the Future

Why we rebuilt LangSmith around the agent
LangChain

Why we rebuilt LangSmith around the agent

From tasks to goals
LangChain

From tasks to goals

More from Together AI

How we make Together AI easy for coding agents to use
Together AI

How we make Together AI easy for coding agents to use

Polymorphic Inference: self-tuning inference for open models
Together AI

Polymorphic Inference: self-tuning inference for open models

Together Compiler: Continuous Compilation for AI inference
Together AI

Together Compiler: Continuous Compilation for AI inference

Auto repair for Together GPU Clusters: from GPU fault to healthy node
Together AI

Auto repair for Together GPU Clusters: from GPU fault to healthy node

Together Post-Training: frontier-grade model shaping, the way that fits you
Together AI

Together Post-Training: frontier-grade model shaping, the way that fits you

Expanding our enterprise inference capacity with IBM Cloud and NVIDIA
Together AI

Expanding our enterprise inference capacity with IBM Cloud and NVIDIA