Hermes Agent Now Caches Tool Schemas on Anthropic - 12K Tokens Stop Getting Re-Sent Every Turn
The largest uncached chunk of data in a tool-heavy Hermes Agent session is not the conversation history. It is the tool schema itself.
On Anthropic's native API, the full tool definitions - names, descriptions, JSON parameter schemas - were being re-sent in full on every internal LLM call. For a session with 40+ tools, the schema alone can exceed 12,000 tokens. Issue #20880 measured the cost: $0.45 for three short turns. For agents that make dozens of internal calls per task, this added up to roughly 70% of input spend being burned on tool definitions the model had already processed.
PR #76032, merged August 1, fixes this. The tool schema now sits inside Anthropic's 4-breakpoint prompt cache budget, alongside the static system prefix and recent conversation endpoints. Every subsequent turn hits cache on the tools.
The 4-breakpoint budget
Anthropic's prompt caching API allows up to four cache_control breakpoints per request. Content up to each breakpoint is cached and reused on subsequent calls with matching prefixes. The budget is fixed - you get four markers, and every byte after the fourth is re-transmitted.
The new cache plan allocates them as follows:
| Breakpoint | Content | Rationale |
|---|---|---|
| 1 | Static system prefix | Never changes per session |
| 2 | Tool definitions (all) | The 12K-token schema - previously uncached |
| 3 | Last completed transaction endpoint | Stable prefix for the current turn |
| 4 | Second-to-last completed endpoint | Covers the previous turn's stable state |
Before this PR, breakpoints 3 and 4 existed but breakpoint 2 was missing. The tool schema sat after the last breakpoint, transmitted fresh on every call.
What changes, technically
The core of the change lives in agent/prompt_caching.py, which gained 156 lines implementing a PromptCachePlan class. The plan is computed per-request and determines which positions in the message array get cache_control markers.
The key constraint: tool markers only fire on direct native Anthropic connections. The gating logic in agent_runtime_helpers.py checks that the resolved provider is api.anthropic.com and the API mode is anthropic_messages. Every other route - OpenRouter, third-party Anthropic-wire gateways, chat_completions - keeps the existing message-only layout unchanged. This is a deliberate safety measure: non-native endpoints may not support tool-level cache markers, or may interpret them differently.
The conversation loop was reworked so that both messages and tools get redecorated on retry. Previously only messages were re-planned after a cache miss; now the full plan regenerates. The agent's canonical tool list is never mutated - a request-local tools_for_api override carries the decorated copy.
For failover, the auxiliary client re-plans both message and tool sections locally against each fallback destination. This means an agent configured with a primary Anthropic endpoint and an OpenRouter fallback gets tool caching on the primary but not on the fallback - and the transition is transparent.
The numbers
| Metric | Value |
|---|---|
| Tool schema size (typical 40-tool agent) | ~12,000 tokens |
| Input spend from tools before fix (3-turn probe) | ~$0.45 |
| Estimated input spend share from tools | ~70% |
| Plan computation cost | 0.41 ms/call |
| Code change | +855 / -57 lines, 13 files |
| Tests | 294 passed, 0 failed (8 new) |
The plan computation overhead is 0.41 ms - effectively zero against the latency of an LLM call. The cache hit eliminates re-transmission of the tool schema entirely, saving roughly 12K tokens of input per turn for a typical tool-heavy agent.
Salvage and safety
This PR is a salvage of #37611 by @rodboev, whose original contribution was cherry-picked with authorship preserved. The adapter wire plumbing for tool cache_control already existed on main (from #23828, in anthropic_adapter.py:1768); this is its first in-tree emitter.
Four adversarial shapes were probed in end-to-end testing: a dangling tool_call with no result, content formatted as a list, and two edge cases around tool result ordering. All four kept the budget invariant - marker count never exceeded four. The prefix stability probe confirmed that the system prefix and shared transaction endpoint markers remain stable across turns, which is the precondition for cache hits.
The non-native fallback path was verified byte-identical to the legacy apply_anthropic_cache_control behavior, with tools left unmarked. No regression for OpenRouter users.
What this means for cost
For an agent session making 20 internal LLM calls with a 12K-token tool schema on Anthropic's Claude:
- Before: 20 calls x 12K tokens = 240K tokens of tool re-transmission
- After: 1 call x 12K tokens = 12K tokens (cached on first use, hits on 19 subsequent)
At Anthropic's Claude Sonnet pricing of $3/M input tokens, that is roughly $0.68 saved per 20-turn session on tool schema alone. For agents running continuously with 100+ turns, the savings compound.
The fix is live on main as of August 1, 2026. Users on direct Anthropic connections get it automatically - no configuration change required.
[^1]: kshitijk4poor. "perf(prompt-caching): cache tool schemas on native Anthropic without history loss." NousResearch/hermes-agent. August 1, 2026.
[^2]: Issue #20880 - measured $0.45 for 3 short turns on tool-heavy agents.
[^3]: rodboev. "Original tool-cache PR #37611." NousResearch/hermes-agent.