Hermes prompt cache hit rate jumps from 0.1% to 98.7% in two PRs

Two PRs merged into Hermes Agent on 19 July 2026 fix the first-call-of-every-turn prompt cache miss. Kshitij K4poor salvaged the work from Soju06's original implementations (#64293 and #64294) in a two-layer stack: an api_content sidecar that persists the exact bytes sent to the API (#67274), and byte-stable system prompts that pin the session-context render across turns (#67403).

Together they shift the prompt cache hit rate from 0.1% to 98.7% on 156k-token contexts. Response time on the first call after a session resume drops from 27.9 seconds to between 2.4 and 5.8 seconds.

Why the cache missed on every turn

Before these PRs, the gateway recomposed the system prompt on every turn. Even when nothing about the session context changed - the thread name, the home directory, the connected platforms - the bytes were different because the render ran fresh each time. The provider's prompt cache saw different content and treated every request as a new one.

Two things broke byte stability. First, the system prompt included per-turn notes: auto-reset notices, first-contact introductions, voice-channel state. These changed every message, so the system prompt always looked different. Second, there was no record of what the previous API call actually sent - the agent composed a message, sent it, and the sent bytes were gone. The next turn had nothing to compare against.

The api_content sidecar (#67274, 16 files, +1,573/-43)

The first PR creates an api_content column on every message row. When the agent sends a message to the provider, it persists the exact bytes - no sanitisation, no stripping - alongside the regular content. On the next turn, the gateway can replay those bytes verbatim.

The implementation touches 16 files. The schema change in hermes_state.py adds a nullable api_content column with declarative auto-migration. The composition in agent/turn_context.py provides a single entry point - compose_user_api_content - for prefetch and plugin injection. Content rewrites that change the message body (context compression, consecutive-user merge, stale-confirmation redaction) drop the sidecar through drop_stale_api_content() so the gateway does not replay stale bytes.

A new test suite at tests/agent/test_api_content_sidecar.py (+1,035 lines, 56 tests) validates the full pipeline.

Byte-stable system prompts (#67403, 6 files, +920/-22)

The second PR builds on the sidecar to freeze the system prompt. It works in three steps.

First, the session-context render is pinned. The gateway computes a sha256 hash over exactly the fields that build_session_context_prompt renders. If the hash matches the previous turn, the pinned bytes are sent verbatim - no re-render. The hash changes only on events that alter the context: a thread rename, a topic edit, /sethome, or a redact_pii flip. On those events, the pin is re-rendered once and re-pinned. The pin is dropped when the cached agent is evicted.

Second, per-turn notes move off the system prompt. Auto-reset notices, first-contact introductions, and voice-channel state now travel on the current user message through the api_content sidecar instead of being appended to the system prompt. They are staged per session, consumed exactly once in run_sync and build_turn_context, and cleared on session finalisation. Multimodal content gets the note appended as a durable text part.

Third, the voice-channel note is emitted only when the state changed since the last delivered value - so repeated silence does not re-trigger the note. A static pointer line in gateway/session.py tells the model that voice state arrives on the message, keeping the model informed without polluting the system prompt.

The test suite at tests/gateway/test_prompt_tail_freeze.py (+410 lines, 37 tests) validates byte stability through live probes: pin verbatim reuse, rename busts once then re-pins, key parity, eviction drops pin, one-shot consume, and multimodal append.

The numbers

Kshitij reported the production impact in the Hermes Discord: on ~156k-token contexts, the first-call prompt-cache hit rate went from 0.1% to 98.7%. Response time dropped from 27.9 seconds to 2.4-5.8 seconds.

Both PRs are on main. They require no configuration changes - the sidecar and the byte-stable pin activate automatically when the gateway runs.

[^1]: Kshitij K4poor. #67274 - feat(cache): persist the exact bytes sent to the API in an api_content sidecar. GitHub. 19 July 2026. [^2]: Kshitij K4poor. #67403 - perf(gateway): byte-stable system prompts - pin the session-context render, deliver per-turn notes on the user message. GitHub. 19 July 2026.