Byte-Stable System Prompts: Hermes Agent Boosts Prompt Cache Hit Rate From 0.1% to 98.7%
A six-file, 920-line PR merged today by kshitijk4poor fixes a long-standing prompt-caching failure in Hermes Agent's gateway: the composed system prompt changed on every single turn, making provider-side caching effectively useless. After the fix, production cache hit rate went from 0.1% to 98.7% on Discord sessions with roughly 156,000-token contexts.
The PR salvages earlier work by Soju06 in #64294, with authorship preserved through cherry-pick.
What was breaking the cache
Most LLM providers offer prompt caching: if a prefix of your request is identical to a previous request's prefix, the provider reuses its cached computation instead of reprocessing those tokens. For a gateway that sends the same system prompt to many conversations, this should produce a high cache hit rate by default.
It did not. The hit rate was 0.1%.
The problem was that Hermes's gateway composed a single system prompt block by concatenating a stable session-context render (agent name, profile, project, tools, skills, memory) with several ephemeral per-turn notes: auto-reset notices, first-contact introductions, and voice-channel state indicators. Every turn, one of these notes changed, which changed the entire composed prompt string, which invalidated the provider's cached prefix.
kshitijk4poor in #67403:
The gateway's composed system prompt is now byte-stable turn-over-turn, so the provider prompt cache and the cached AIAgent survive across messages instead of being re-keyed/rebuilt on every one.
The fix: pin, separate, sort
The PR addresses the problem in three layers, each touching a different structural cause of cache invalidation:
| Layer | Problem | Fix |
|---|---|---|
| Session-context pinning | System prompt re-rendered on every turn, producing different byte sequences even when nothing changed | Render once per session, pin as bytes, reuse until a cache-busting event (thread rename, topic edit, /sethome, redact_pii) forces re-render |
| Per-turn note separation | Auto-reset notices, first-contact intros, and voice-channel state were baked into the system prompt, changing it on every turn | Notes moved to the user message via the api_content sidecar, consumed exactly once, and cleared on session finalization |
| Deterministic platform ordering | Restart or registration order could reorder the list of connected platforms in the system prompt, changing its byte sequence | get_connected_platforms() now sorts its output so the rendered platform list is stable regardless of startup order |
The pinning mechanism uses a SHA-256 hash computed over exactly the fields that build_session_context_prompt renders. On each turn, the gateway hashes those fields. If the hash matches the stored key, it reuses the pinned bytes verbatim. If the hash differs (cache-busting event), it re-renders once and pins the new result.
File-level breakdown
| File | +Lines | -Lines | Purpose |
|---|---|---|---|
| gateway/run.py | +221 | -20 | Per-session pinning of the session-context render, note delivery to user message, voice-channel state tracking |
| agent/turn_context.py | +63 | -0 | One-shot note consumption in turn context builder, multimodal message appending |
| gateway/config.py | +9 | -2 | Sorted get_connected_platforms() for deterministic output |
| gateway/session.py | +10 | -0 | Static pointer line telling the model voice state is on the user message |
| tests/gateway/test_prompt_tail_freeze.py | +410 | -0 | 37 tail-freeze test cases verifying the pinned prompt does not drift |
| tests/agent/test_gateway_turn_sidecar.py | +207 | -0 | 8 turn-sidecar tests verifying one-shot note delivery and consumption |
| Total (6 files) | +920 | -22 |
The test suite is the story's safety net. The 37 tail-freeze tests verify that the pinned prompt bytes do not drift across turn sequences, including edge cases like concurrent thread renames and rapid-fire /sethome calls. The eight turn-sidecar tests verify that per-turn notes are delivered exactly once and never leak between sessions. An adversarial key-collision probe confirmed that repr(tuple) escaping cannot be spoofed to produce hash collisions with manipulated chat names.
Production impact
The PR body reports Discord production numbers measured by the contributor with approximately 156,000-token contexts:
| Metric | Before | After |
|---|---|---|
| Prompt cache hit rate | 0.1% | 98.7% |
A 98.6 percentage point improvement. For a 156,000-token context on a provider that charges for cached vs. uncached input tokens at different rates, the cost difference per turn is substantial. Every turn where the system prompt was being re-processed unnecessarily -- roughly 999 out of every 1,000 turns before the fix -- is now a cache hit.
What this means downstream
This optimization has no user-facing feature change. No new commands, no new UI. It changes how the gateway assembles its outbound prompt so that the parts that do not change between turns genuinely do not change at the byte level. The result is that providers like Anthropic (Claude), which charge 90% less for cache-hit input tokens than cache-miss tokens, now see nearly every system prompt as a cache hit.
For operators running Hermes Agent through a gateway with multiple concurrent conversations, the cost of the system prompt portion of each API call effectively drops to one-tenth of what it was yesterday.
[^1]: kshitijk4poor. "perf(gateway): byte-stable system prompts - pin the session-context render, deliver per-turn notes on the user message." Hermes Agent PR #67403. July 19, 2026.
[^2]: Soju06. "Original byte-stable system prompt PR." Hermes Agent PR #64294. Salvaged by kshitijk4poor in #67403.