Our top agent latency optimisations

Blog/Research/Agent latency

Blog/Research/Agent latency

A model can emit a complete tool call while it is still streaming later calls or text. Waiting for the entire provider response turns two independently useful clocks into one long serial clock.

How it works

Settle each locally executable tool call into a supervised task as soon as its complete event arrives. More tool calls and the remaining provider stream can continue concurrently. A one-permit publication gate protects ordered history while expensive execution remains parallel, and the runner joins outstanding work only after stream settlement.

The key separation is between execution readiness and transcript readiness. The former can be eager; the latter must retain the exact order expected by replay and the next model request.

The Improvement

If a shell command takes eight seconds and the model needs another five seconds to finish emitting calls, serial execution costs roughly thirteen seconds. Starting the shell immediately exposes only the slower of the two clocks, plus small coordination overhead.

The improvement compounds when the response contains several independent calls, especially searches, reads and remote API requests with variable latency.

How to action it

Dispatch at the completed tool-call boundary, not at the end of the whole provider message:

• Parse incrementally: consume stream events as they arrive and maintain one argument accumulator for each tool-call identity.
• Use an explicit readiness event: a tool becomes executable only when its block closes, not when the first argument fragment appears.
• Validate before execution: parse the accumulated payload, apply the authoritative schema and reject partial or invalid input rather than guessing.

Read more

E2E latency ↓ 10%

Microsoft Research’s SUTRADHARA dispatches tool calls incrementally during model decoding instead of waiting for the complete output. Its combined design reduced median first-token-rendered latency by 15% and end-to-end latency by 10%; those figures are not an isolated streaming-dispatch result. Microsoft Research: SUTRADHARA → Anthropic’s implementation guide specifies the safe parser contract: accumulate input deltas, parse when the tool block closes and reject invalid input. Anthropic tool streaming →

Guardrail

Do not publish nondeterministic completion order into model history. Define cancellation behavior for tools still running when the stream fails, bound progress-event backpressure, and start only calls whose arguments are complete and validated.

Metrics

tool-ready-to-startstream/tool overlap msuncovered tool timeorphaned-tool count

Agent loops repeatedly call the same model with a mostly unchanged conversation. Treating every continuation as a brand-new HTTP request forces the surrounding API stack to rediscover and reprocess state that was valid milliseconds ago.

continuation cost = new input processing + incremental inference, not full connection + full history processing

How it works

Keep a session-scoped streaming connection available across model steps. When non-input request properties are unchanged and the next input strictly extends the prior exchange, the client sends only the new tail with the provider-issued previous-response identity. Server-side state can then reuse prior items, tool definitions, routing decisions and rendered-token artifacts.

Retain idle connections for a bounded window and avoid head-of-line blocking: if the cached connection is busy, another request can use a temporary connection.

The Improvement

The gain repeats on every tool continuation. It removes connection establishment and prevents conversation-length-dependent request processing from growing throughout the run. This matters increasingly as model inference becomes faster and API/service overhead becomes a larger share of end-to-end time.

The optimisation is structural: it preserves the familiar request shape while making the underlying stack incremental.

How to action it

Treat continuation state as a leased, fingerprinted session resource:

• Keep one hot transport per active session: reuse a streaming connection across the model–tool loop instead of reconnecting for every continuation.
• Store the continuation identity: retain the provider’s previous-response identity alongside a fingerprint of the model, instructions, tools, sampling settings and routing properties.
• Send only a strict tail: use incremental continuation only when the new input extends the prior exchange and the non-input fingerprint is unchanged.

Read more

Up to 40% faster

OpenAI reports up to 40% faster agentic workflows after moving Responses continuations onto WebSockets with connection-scoped state. It attributes the result to incremental validation, token rendering, routing reuse and overlapping non-blocking post-inference work. OpenAI WebSocket engineering →

Guardrail

Delta continuation is safe only while model, tools, system configuration and routing identity remain compatible. After a partial or ambiguous transport failure, do not blindly replay work that may already have executed; clear continuation state or reconcile it first.

Metrics

continuation TTFTbytes per continuationsocket reuse ratefull-context fallback rate

Small prompts are not automatically fast prompts. If withholding predictable task context forces the model through discovery, search and retrieval loops, a larger first request can finish sooner overall.

use more initial context only when it removes later model/tool round trips or improves cache reuse

How it works

Include the task inputs the model will almost certainly need in the initial request when they fit a conservative context threshold. These inputs may include relevant files, records, schemas, policies or application state. The model can begin useful work without first calling tools merely to discover and retrieve known inputs. For oversized sources, provide a bounded, addressable index with stable identities, sizes and short previews.

Place stable instructions, tool definitions and shared reference context before rapidly changing history or retrieval results so providers can reuse the longest common prefix.

The Improvement

The win is not “fewer input tokens”; this path may use more. It wins when one cached prefill replaces multiple sequential inference and tool phases. It also reduces zero-progress starts in which the model spends its first response locating context the host already knew it would need.

A threshold keeps the strategy bounded, while an indexed fallback preserves orientation across large or distributed sources.

How to action it

Build a context policy around total task time, not the smallest possible first prompt:

• Identify predictable context: inspect traces for inputs the model nearly always retrieves before it can make its first useful decision.
• Set an inclusion threshold: provide those inputs on the first call when they fit while preserving explicit headroom for reasoning, tool results and the answer.
• Use an addressable index: for oversized or distributed sources, provide stable item identities, sizes and short previews plus tools for selective retrieval.

Read more

Context is conditional

OpenAI’s guide estimates that halving ordinary input may improve latency by only 1–5%, while fewer requests can remove whole round trips. It also recommends placing dynamic material later to maximise a shared cacheable prefix. OpenAI latency optimisation →

Guardrail

Do not inline stale, unauthorised or irrelevant content. Track source versions, preserve the provider’s real context headroom, and A/B test total completion time rather than just time to first token. Bigger context can reduce attention quality even when it fits.

Metrics

first-run tool counttime to first useful actionprompt-cache hit ratetotal run tokens

Models are good at writing small programs. Letting a program inspect, filter, transform and validate local data can replace a long sequence of model → tool → model round trips.

many tool continuations → one model decision + local compute + one bounded receipt

How it works

Give the model one restricted programming interface over an immutable working snapshot. The program can locate sections, preview replacements and construct a versioned patch plan or grouped bundle. Large snapshots and intermediate values stay inside the sandbox; only a bounded result and canonical staging receipt return to the transcript.

The sandbox cannot apply edits. After successful process exit, the host validates document and snapshot identity, plan structure, permissions, review rules and resource limits, then invokes the ordinary canonical proposal path once.

The Improvement

Local loops, joins, filtering and string transforms run at ordinary program speed and do not consume model context between each step. One call can stage a bounded group of edits, while strict compute and result-size limits cap pathological work.

This pattern is especially strong when tool results are large but the final decision needs only a small aggregate.

How to action it

Move a repeated tool chain into code only when its intermediate values do not need fresh model judgement:

• Choose the right traces: target workflows dominated by fetch–filter–join–transform loops or large values copied between tools.
• Expose narrow typed APIs: make the available operations discoverable inside the execution environment and load only the definitions the task uses.
• Keep control flow local: perform loops, conditionals, retries and safe parallel calls in the program instead of alternating through the model after every step.

Read more

150k → 2k tokens

In Anthropic’s worked MCP example, exposing tools as discoverable code APIs and processing data in the execution environment reduced tool-related context from 150,000 to 2,000 tokens, a 98.7% reduction. That is an example, not a universal result. Code execution with MCP →

Guardrail

Code execution trades conversational simplicity for a serious security boundary. Use an immutable input snapshot, no ambient credentials, strict CPU/memory/time/output limits, versioned schemas and a separate host-authorised commit phase. Never infer that a successful process exit means its proposed effects are safe.

Metrics

model invocations per edit tasktool calls per tasksandbox/host time splitplan rejection rate

When one model response contains independent searches, reads or calculations, sequential execution pays the sum of every duration. Parallel execution pays approximately the slowest duration, but only if ordering and side effects are designed explicitly.

sequential batch: Σ Ti → parallel batch: max(Ti) + coordination

How it works

Execute a batch concurrently by default while preserving a global sequential mode and a per-tool serial escape hatch. The safety constraint therefore sits next to the tool definition that understands its own side effects.

Each parallel tool can publish completion as soon as it finishes, while tool-result messages are appended to model history in original call order after the batch joins. User feedback and deterministic context use different clocks.

The Improvement

Remote searches and file reads have variable latency, so concurrency also reduces exposure to additive variance. The pattern is more robust than hard-coded lists because new tools inherit parallel behavior unless they explicitly opt into serialization.

Even nominally sequential steps can sometimes use cancellable speculative execution when one branch is overwhelmingly likely.

How to action it

Define dependency and side-effect policy before enabling concurrency:

• Accept real batches: parse every tool call in one model response and retain each call identity and original position.
• Classify execution safety: run independent reads and commutative calculations concurrently; mark mutations, shared-state operations and ordered workflows as serial.
• Bound the batch: use a concurrency limit, per-tool timeout and shared cancellation policy rather than launching unbounded work.

Read more

Parallel + speculative

OpenAI’s guide recommends parallel calls for non-dependent steps and cancellable speculative work when a likely branch can start before an earlier classification completes. OpenAI latency optimisation → Anthropic’s tool-use guide adds the execution rule: parallelise independent read-only calls, serialise side effects or ordering dependencies and return every result together with its call identity. Anthropic parallel tool use →

Guardrail

Parallelise only calls that do not race on shared state. File mutations often need ordering by canonical path; financial, permission-changing and externally visible actions may need a stronger transaction or global serial lane. Preserve model-call order in durable history even if UI completion is eager.

Metrics

batch sum vs wall timepeak tool concurrencyserial fallback rateordering violation count

An agent with 500 tools should not pay for 500 full schemas on every model call. Nor should a 20 MB command result become permanent transcript baggage when the model needs only its tail and a way to inspect more.

small searchable catalogue → hydrate selected schema/result → preserve address for later reads

How it works

Represent deferred tools with compact searchable metadata, index the catalogue locally and expose only matching definitions to the next model call. Reintroduce a deferred definition at the transcript position that activated it, avoiding a permanently inflated initial tool prefix.

The same principle applies to output: keep a bounded preview or rolling tail in context, spill the full value to a file or durable object and return its address. The agent can progressively read more if the task proves it is necessary.

The Improvement

Smaller tool prefixes reduce token processing, ambiguity and cache-prefix churn. Addressable results prevent a single noisy command from forcing compaction and make summarisation less destructive because the underlying evidence still exists outside the summary.

This differs from optimisation five: code mode compresses a compound workflow; progressive disclosure controls what enters model context at all.

How to action it

Replace always-on context with compact catalogues and durable addresses:

• Publish a small catalogue: give the model tool names, short descriptions, grouping and availability status instead of every full schema.
• Hydrate on selection: let the agent search or browse the catalogue, then load the exact authoritative schema before execution and keep activated definitions in working context.
• Spill rather than truncate: write long tool output, terminal history and prior transcript detail to a durable object; return a bounded preview or tail plus its address.

Read more

46.9% fewer tokens

Cursor reports a statistically significant 46.9% reduction in total agent tokens, with high variance, for MCP-using runs after loading tool descriptions dynamically. It also stores long outputs as files so the agent can inspect them on demand. Cursor dynamic context discovery →

Guardrail

Discovery introduces a recall problem. Always hydrate the authoritative schema before execution, evaluate whether search hides necessary tools, retain stable addresses for spilled output and make truncation explicit. “Not in context” must not silently become “does not exist.”

Metrics

tool-schema tokens/requesttool-search recallforced compactionsspilled-result reread rate

The fastest setup is often setup that the current task never waits for. Late binding, background refresh and speculative warm-up attack cold latency more effectively than making every dependency initialize slightly faster.

first response depends only on required-now state; optional execution environments and catalogues attach later

How it works

Return the provider event stream before dynamically importing the selected implementation so unused providers never join the eager module graph. Paint the interface from the last-known model snapshot before remote catalogue refresh starts, and perform best-effort transport warm-up while session startup continues with a normal timeout fallback.

The strongest form is architectural late binding: keep inference orchestration independent from sandboxes, repositories and tools, then provision a “hand” only when a tool call actually needs it.

The Improvement

Optional dependencies no longer contaminate every request’s TTFT with their p95. The common no-sandbox or no-refresh path starts immediately, while likely future work can still be warmed speculatively behind visible progress.

This also improves fault isolation: a catalogue outage or sandbox delay does not have to prevent the model from acknowledging and reasoning about the request.

How to action it

Draw the startup dependency graph and move everything not required for the first token behind that boundary:

• Label required-now work: keep only admission, authorisation, durable session recovery and the selected model path on the blocking startup chain.
• Provision hands on demand: start inference before cloning repositories or creating sandboxes, then provision an execution environment only when a tool call actually needs one.
• Load providers lazily: dynamically import the chosen implementation after the stream contract exists so unused integrations never enter the cold path.

Read more

p50 ↓ \~60%; p95 ↓ >90%

Anthropic reports roughly 60% lower p50 TTFT and over 90% lower p95 after decoupling the agent “brain” from execution containers, allowing inference to start before a sandbox is provisioned. Scaling Managed Agents →

Guardrail

Background work still needs ownership, cancellation and bounded failure handling. Warm-up must be best effort, never a new mandatory gate. Lazy-import errors should close the ordinary stream contract cleanly, and stale local catalogues must be labelled or reconciled before tool execution.

Metrics

cold TTFT p50/p95first paintsetup actually usedwarm-up hit rate

Double-clicks, network retries and worker failover are normal. Without a durable identity, each can launch another complete agent run or provider request. That doubles cost and often making latency worse through self-inflicted contention.

same logical request + same identity → return or resume original work, never start a sibling

How it works

Hash the logical request after removing server-assigned runtime identifiers. Key a durable admission record by session, user and client request identity. The first caller owns admission; a duplicate with the same fingerprint waits for or returns the stored response. Reusing the key for different content produces a conflict rather than accidental aliasing.

If the admission record is still pending but its run already exists, reconstruct the accepted response from that run. Claims can be reassigned only before they bind to a durable run.

The Improvement

Retry traffic becomes a database lookup instead of another model loop. This is primarily a tail and overload optimisation: duplicate work is rare in healthy conditions but expensive during the exact incidents that already stretch response times.

At the provider boundary, separately hash canonical conversation messages, transport-decorated messages and compiled provider context. These content identities create an auditable pre-transport fence and stable recovery key.

How to action it

Make logical request identity an explicit API contract rather than guessing from similar payloads:

• Require an idempotency key: use a caller-generated identity scoped to the tenant, session and logical operation, and reuse it for every retry.
• Record intent before work: atomically store the key, a normalised request fingerprint, ownership, status and eventual result before launching expensive execution.
• Reuse exact duplicates: when key and fingerprint match, wait for, resume or return the original run and a semantically equivalent response; never start a sibling.

Read more

At-most-once work

AWS’s Builders’ Library explains how a caller-provided request identifier lets a service recognise retries, avoid repeating side effects and return a semantically equivalent response. It also requires atomic recording with the mutation and treats reuse of one identifier with different parameters as a validation error. The same ownership and conflict boundaries this optimisation needs. AWS Builders’ Library →

Guardrail

Scope identities by tenant and session, fingerprint all semantics that can change the result, retain responses long enough for realistic retry windows and never “take over” work already bound to a durable execution. Design separate rules for safe pre-send retries and ambiguous post-send failures.

Metrics

duplicate runs preventedadmission reuse rateidentity conflictsambiguous retry count

Token streaming can make the model feel fast while downstream systems quietly serialize hundreds of writes, notifications and terminal redraws. The durable and visual pipeline must be designed as part of the same latency budget.

coalesce replaceable updates; preserve explicit durability, close and replay barriers

How it works

At a candidate terminal boundary, keep item snapshots and terminal events staged until the runtime knows whether another invocation is required. A true terminal run commits authoritative items, lifecycle metadata and the closing run snapshot in one business transaction. First and last outbox fences prove delivery of the contiguous prefix, and one high-watermark stream wake tells the client to read through the complete ledger cursor.

At the display layer, coalesce consecutive commits sharing semantic identity into one queued update. In terminal interfaces, limit render invalidations to a frame budget, scan only the visible viewport and write only the changed line range.

The Improvement

Per-item internal HTTP calls, database round trips and full-screen writes no longer grow linearly with event volume. More importantly, coalescing reduces the number of opportunities for a single slow component to dominate final perceived latency.

Semantic identity is stronger than a timer alone: it says which updates can safely replace or append to one another and where the system must flush.

How to action it

Coalesce separately at the durability, notification and rendering layers:

• Define semantic identity first: document which updates supersede one another and which events such as commit, close, cancellation, destroy and replay movement are mandatory flush barriers.
• Bundle terminal durability: write final items, lifecycle state and an ordered outbox record in one transaction so rollback cannot emit a false notification.
• Wake by watermark: publish one sequence or high-watermark notification and let idempotent consumers read the complete committed range in order.

Read more

Optimise the tail

Google’s “The Tail at Scale” explains why rare slow components dominate as systems fan out. Google Research → AWS documents atomically writing state and ordered outbox events, with idempotent consumers. AWS transactional outbox → MDN recommends batching DOM changes and scheduling visual work for the browser’s repaint cycle. MDN JavaScript performance →

Guardrail

Never coalesce away distinct durable facts. Terminal commit, close, destroy, cancellation and replay-cursor transitions are flush barriers. A wake may be replaceable only when its cursor covers every omitted notification; a visual update may be dropped only when newer state fully supersedes it.

Metrics

provider-stop-to-settled UIwrites per runSSE wakes per terminal bundlerendered lines per frame

Making a model call 10% faster is useful. Proving that the call is unnecessary removes its queueing, network, prefill, reasoning and generation time together. This is why terminal control flow outranks almost every lower-level optimisation.

saved latency ≈ provider admission + network + prefill + reasoning + output generation

How it works

A terminal tool must change host-owned state, not merely ask the model to stop. One terminal state can mark the run complete; another can store a structured pending request for user input. After the durable tool result is appended, the loop checks those states and exits without a ceremonial wrap-up invocation.

The same approach works after an edit. For a mechanical request, a conservative classifier can let the server confirm that the change was staged for review. If the user also needs a substantive answer, the execution tool returns a model-written draft, which the host releases only after the edit plan commits.

The Improvement

These paths remove serial work; they do not just hide it. The same rule applies at a compaction boundary: when a new user prompt triggers pre-prompt compaction, the imminent user message is already the continuation. No empty model call should be inserted between compaction and that prompt.

The saving grows with slow models, large cached or uncached prefixes, provider congestion and long generated summaries. Those are exactly the conditions in which a redundant invocation hurts most.

How to action it

Give every model invocation a reason code, then target these recurring cases:

• Terminal tools: make the tool result explicitly terminal. Once the result is durably recorded, close the run and publish the known outcome without asking the model to narrate it.
• Mechanical confirmations: render fixed messages such as “changes staged” or “export complete” from a validated receipt.
• Commit-gated drafts: let the original model response include bounded user-facing text alongside its tool call. Publish that draft only after the transaction succeeds; discard it on failure.

Read more

One fewer request

OpenAI’s latency guide ranks “make fewer requests” as a core principle because every request adds a round trip. It also recommends replacing constrained model outputs with deterministic application logic when possible. OpenAI latency optimisation →

Guardrail

Only terminate when the host can prove the semantic obligation is complete. Mechanical completion must fail closed for requests asking for analysis, risk, alternatives, citations or explanation. Commit-gated drafts must be discarded when validation or staging fails.

Metrics

model\_invocation\_countterminal-tool-to-run-completepost-edit continuation rateincorrect early-exit rate

Previous

Next