How Hermes Agent Handles Long Conversations: Dual-Layer Compression and Anthropic Prompt Caching

Agents that run continuously accumulate a problem. Every tool call, every file read, every terminal output gets fed back into the conversation. Over hours or days, the message list grows until it hits the model's context limit. At that point the agent either fails with an API error or silently loses the beginning of the conversation.

Hermes Agent addresses this with two independent systems: a dual-layer compression pipeline that fires at different thresholds, and an Anthropic prompt caching strategy that avoids re-reading the conversation prefix on every turn. HZ0108 published a detailed source-code analysis of Hermes's architecture in May 2026, and the official Hermes documentation confirms the algorithms. Here is how both systems work, with the actual numbers.

Dual-Layer Compression

Hermes runs two compressors at different points in the pipeline, each with its own threshold. They never coordinate -- each operates independently.

The gateway hygiene layer is the safety net. It fires before the agent processes a message, only when the history has at least 4 messages and compression is enabled. Its 85% threshold is intentionally higher than the agent compressor. As the docs note, setting it at 50% "caused premature compression on every turn in long gateway sessions."

The agent compressor is the primary system. It runs inside the agent's tool loop with access to accurate, API-reported token counts. The default threshold is 50% of the model's context window, configurable via compression.threshold in config.yaml.

For a 200K context model at default settings:

context_length       = 200,000
threshold_tokens     = 200,000 x 0.50 = 100,000
tail_token_budget    = 100,000 x 0.20 = 20,000
max_summary_tokens   = min(200,000 x 0.05, 12,000) = 10,000

The threshold is always derived from the main agent model's context window, not the auxiliary model used for summarization.

4-Phase Compression Algorithm

When compression triggers, it runs through four phases:

Phase 1: Prune Old Tool Results

Tool outputs over 200 characters that sit outside the protected tail region are replaced with a placeholder. This is a cheap pre-pass -- no LLM call -- that clears verbose terminal output, file contents, and search results.

Phase 2: Determine Boundaries

The message list is divided into three regions. The first few messages (system prompt plus first exchange, 3 messages hardcoded) are always preserved. The tail is protected by token budget: the compressor walks backward from the end, accumulating tokens until the budget (threshold_tokens x target_ratio) is exhausted. If the budget would protect fewer than protect_last_n messages (default 20), the fixed count wins.

Boundaries are aligned to avoid splitting tool_call/tool_result groups. The compressor walks past consecutive tool results to find the parent assistant message, keeping groups intact.

Phase 3: Generate Structured Summary

The middle section is sent to an auxiliary LLM with a structured template. The summary covers: goal, constraints, progress (done, in progress, blocked), key decisions, relevant files, next steps, and critical context.

Summary budget scales with content: content_tokens x 0.20, floored at 2,000 tokens and capped at min(context_length x 0.05, 12,000).

The summary model must have a context window at least as large as the main model's. If it does not, the API returns a context-length error, the compressor falls back to dropping middle turns without a summary, and conversation context is silently lost.

Phase 4: Assemble Compressed Messages

The output message list is: head messages (with a note appended to the system prompt on first compression), a summary message, then tail messages. Orphaned tool_call/tool_result pairs are cleaned up -- results referencing removed calls are dropped, and calls whose results were removed get a stub injected.

Iterative Re-compression

On subsequent compressions, the previous summary is passed to the LLM with instructions to update it rather than summarize from scratch. Items move from "In Progress" to "Done," new progress is added, and obsolete information is removed. This prevents signal degradation from multi-round compression -- the "summary of a summary of a summary" problem.

Before and After

The documentation provides a concrete example. A 45-message conversation at ~95K tokens compresses to 25 messages at ~45K tokens:

The compressed conversation retains the project structure summary (goals, completed work, in-progress items, relevant files, next steps) while dropping the individual edit/read cycles from the middle. The system prompt and recent turns survive intact.

Prompt Caching: system_and_3

Separate from compression, Hermes implements Anthropic prompt caching to avoid paying full input-token prices on every turn. Anthropic allows up to 4 cache_control breakpoints per request. Hermes uses the system_and_3 strategy:

Breakpoint 1: System prompt           (stable across all turns)
Breakpoint 2: 3rd-to-last non-system message  --|
Breakpoint 3: 2nd-to-last non-system message    |-- Rolling window
Breakpoint 4: Last non-system message          --|

This reduces input token costs by ~75% on multi-turn conversations. The recent user message is intentionally not cached -- this supports parallel agent forking scenarios where multiple agents share the cached prefix but diverge on the most recent turn.

Prompt caching is automatically enabled when the model is Anthropic Claude and the provider supports cache_control (native Anthropic API or OpenRouter). The cache TTL defaults to 5 minutes, configurable to 1 hour for sessions where the user takes breaks between turns.

Cache-Aware Design

Several design decisions make caching reliable:

• The system prompt is breakpoint 1 and cached across all turns. Compression appends a note only on the first compaction, preserving stability.
• Cache hits require prefix matching. Adding or removing messages in the middle invalidates the cache for everything after.
• After compression, the cache is invalidated for the compressed region but the system prompt cache survives. The rolling 3-message window re-establishes caching within 1-2 turns.
• Model identity is part of the cache key. Mid-conversation model switches or credential rotations mean zero cache hits on the next request.

How They Work Together

Compression and caching serve different problems. Compression shrinks the message list so the agent stays within context limits. Caching avoids re-reading the stable prefix on every turn. They interact cleanly: after compression, the system prompt cache survives, and the rolling window of recent messages re-establishes caching quickly.

The dual-layer design (50% agent compressor + 85% gateway hygiene) means most sessions are handled by the agent compressor with accurate token counts. The gateway hygiene layer catches edge cases -- sessions that grew too large between turns, overnight accumulation on messaging platforms, or sessions where the agent compressor was disabled or misconfigured.

Configuration

All compression settings live under compression in config.yaml:

compression:
  enabled: true
  threshold: 0.50
  target_ratio: 0.20
  protect_last_n: 20

Prompt caching has one knob:

prompt_caching:
  cache_ttl: "5m"

The CLI confirms both systems at startup:

💾 Prompt caching: ENABLED (Claude via OpenRouter, 5m TTL)

---

The source code analysis by HZ0108 is available at hermes-agent-insights. The compression and caching architecture is documented in the Hermes Agent developer guide.

[^1]: HZ0108. "Inside Hermes Agent: Architecture and Design Philosophy." GitHub. May 2026. [^2]: Nous Research. "Context Compression and Caching." Hermes Agent documentation.