Making open models work for long-horizon agentic workloads

Practical AI: Tools, Models & Frameworksinference

What changed

Ollama and Together AI published a detailed guide on optimizing open models for long-horizon agentic workloads, covering both harness-side (request construction) and inference engine-side (caching) strategies. Key recommendations include: placing dynamically changing content in task messages rather than system prompts, loading tool definitions at the end of prompts rather than the top, truncating tool outputs with explicit range indicators, and routing subsequent agent turns to the same node to preserve prefix cache. The post explains how prefix caching works across agent turns and why a single character change early in the prompt forces reprocessing of all subsequent tokens.

Why it matters

Agent sessions grow quadratically in compute cost without prefix caching (1+2+...+N tokens) but linearly with it (N tokens). For hundred-turn coding tasks, this difference is 5,050 versus 100 units of work. Many common harness patterns—timestamped system prompts, changing reasoning effort mid-session, poorly truncated tool results—silently break the cache, adding substantial latency and cost. Following these patterns lets builders use open models economically on tasks previously only practical with frontier APIs.

How it compares

Unlike prior work on individual open models (DeepSeek-V3, GLM-5.3), this post bridges the gap between harness design and inference infrastructure, showing how architectural choices on both sides determine whether prefix caching actually works. It treats agent workflows as a distinct pattern requiring different optimization than single-turn chat or reasoning tasks.

Sources