Hermes Kanban SQLite WAL Hardening - Six Commits That Stopped Database Corruption Under Concurrent Dispatchers

Hermes Agent v0.18 landed a coordinated set of Kanban SQLite fixes between May 23 and June 12, 2026, closing eight corruption issues that had stacked up across April and May. The surface is ~/.hermes/kanban.db and the failure modes are textbook SQLite-under-load: WAL sidecar unlink races, torn writes, immediate SQLITE_BUSY aborts, and spurious corrupt-DB quarantines. Six commits harden the connection layer. The work is unglamorous and the failure modes are specific, but the same pattern applies to any SQLite-backed agent that runs multiple processes against one DB.

@lemondooe framed the underlying announcement well on July 4 in a quote-tweet of Nous Research's June 29 database concurrency update:

Hermes Agent concurrency updates look boring from the outside. That is exactly why they matter. Once you actually run multi-agent work, the problem stops being "what prompt should I use?" It becomes state, queues, database locks, context isolation, task boards, retries and...

The original Nous Research tweet announcing the concurrency hardening crossed 747 likes, 50 retweets, and 105K impressions inside a day. The link target in that tweet points to the upstream fix cluster that closed issues #31502, #32424, #32532, #33169, and #33334.

The failure pattern

Hermes runs multiple processes against one SQLite DB by design: the gateway dispatcher, the dashboard, CLI sessions, profile-specific gateways (--profile bingge, --profile pixiel, --profile mafei), and Kanban workers all share ~/.hermes/kanban.db and ~/.hermes/state.db. SQLite's WAL journal mode permits concurrent readers and a single writer per connection, but it does not protect against the failure modes that show up when several processes open the same file in parallel and each runs its own connection lifecycle.

The corruption pattern that surfaced across four issues in May had three specific triggers:

WAL sidecar unlink on every connect. kb.connect() ran PRAGMA journal_mode=WAL on every call. SQLite treats that as a fresh-setup call and unlinks/recreates the -wal and -shm sidecars even when the DB is already in WAL mode. With four profile gateways concurrently opening connections, the unlink-recreate race against any other process's active WAL write was the mechanism producing the btreeInitPage() returns error code 11 and wrong # of entries in index ... pattern that users reported.

Immediate SQLITE_BUSY under concurrent writes. Two processes attempting simultaneous writes received immediate SQLITE_BUSY from SQLite, aborted mid-transaction, and tore the WAL. The fix is a busy_timeout PRAGMA so concurrent writers wait via SQLite's lock queue instead of aborting.

Transient EIO misclassified as WAL-incompatible. _WAL_INCOMPAT_MARKERS previously matched the substring disk i/o error. A transient I/O error during the WAL pragma was misclassified as a permanent WAL-incompatible-filesystem condition and triggered a second mutating fallback: PRAGMA journal_mode=DELETE. That second fallback was the next failure mode that hit the same boards.

The six commits

The hardening landed across six commits between May 23 and June 12:

The first commit is the load-bearing one. Without the read-only probe, every other PRAGMA change still races against the WAL sidecar churn. With it, the sidecar churn stops on the multi-process open path and the rest of the defenses stack cleanly on top.

The PRAGMA stack on a healthy connection

After the hardening lands, kb.connect() runs this sequence:

PRAGMA journal_mode=WAL          # only if probe says mode != 'wal'
PRAGMA synchronous=FULL          # fsync before each checkpoint
PRAGMA wal_autocheckpoint=100    # tight checkpoints, ~400KB / checkpoint
PRAGMA secure_delete=ON          # zero freed pages
PRAGMA cell_size_check=ON        # corrupt cells surface as read errors
PRAGMA busy_timeout=120000       # 120s wait on SQLITE_BUSY

The session storage layer (state.db) applies the same PRAGMA stack via apply_wal_with_fallback() plus application-level retry with random jitter (20-150ms, up to 15 retries) and BEGIN IMMEDIATE transactions to surface lock contention at transaction start. The retry logic addresses the "convoy effect" where SQLite's deterministic internal backoff causes all competing writers to retry at the same intervals.

Why this matters for multi-agent Kanban runs

The Kanban board is the substrate for Hermes's multi-agent dispatch. When a dispatcher spawns five worker tasks against the same board, each task gets its own connection that opens and closes against kanban.db repeatedly across its lifetime. Without busy_timeout, the moment two workers hit the DB in the same millisecond, one of them gets SQLITE_BUSY, aborts mid-transaction, and the WAL frame is torn. Without the WAL sidecar fix, every connect across the swarm races the active writer's sidecar. The hardened PRAGMA stack means each connect waits its turn via the SQLite lock queue, the sidecars stay put, and torn writes either complete cleanly or fail loudly at the post-commit invariant check.

Issue #32532 framed the original busy_timeout proposal:

kanban_db.py initializes SQLite with WAL journal mode and synchronous=NORMAL, but does NOT set busy_timeout. When two processes (Gateway Dispatcher + Dashboard Plugin) attempt simultaneous writes, SQLite returns SQLITE_BUSY instead of waiting. This causes WAL journal corruption.

The fix landed as commit 90b6b3d18 with a more generous 120s timeout than the 5s originally proposed. The full hardening pattern - read-only WAL probe, narrowed incompatibility markers, busy_timeout, torn-write pragmas, post-commit invariants, time-bounded quarantine - is now the upstream baseline.

Beyond SQLite: PostgreSQL option

For deployments where SQLite's single-writer model is structurally insufficient, PR #39301 adds an opt-in PostgreSQL backend for state.db. The migration path includes dry-run/apply modes, streaming import, row-count verification, and DSN redaction. A test migration validated 5,213 sessions and 172,847 messages copied to PostgreSQL with exact count preservation. The architecture supports both backends through the same SessionDB API surface; SQLite remains the default.

The Kanban board itself is still SQLite-only - there is no PostgreSQL backend for kanban.db yet. For users running heavy concurrent dispatch, the SQLite hardening series is the answer; for users who need real concurrent writers, the PostgreSQL state backend is the next step.

[^1]: @lemondooe. "Quote-tweet of Nous Research database concurrency update." July 4, 2026.

[^2]: Nous Research. "Hermes Agent now has no problem with scaling huge multi-agent kanban runs." June 29, 2026.

[^3]: @steveonjava. "PR #32532 - add busy_timeout PRAGMA to prevent WAL corruption under concurrent writers." Hermes Agent. Closed.

[^4]: @steveonjava. "PR #33334 - Database corruption with Kanban causing system crash." Hermes Agent. Closed.

[^5]: @jamesraddock. "PR #41795 - stop corruption amplification + spurious quarantines in the DB health guard." Hermes Agent.

[^6]: Hermes Agent. "Session storage developer guide."

[^7]: PR #39301. "feat: add PostgreSQL state backend." Hermes Agent.