2026-07-05

Hermes Agent's Record & Replay skill captures macOS workflows via CGEventTap, generates replayable skills

hermesskillscomputer-userecord-replayagent-design

Hermes Agent now learns to repeat your boring tasks after watching once

Show it the sequence. It saves the whole workflow. Next time it runs start to finish on its own.
Same clicks. Same order. Every time. No more repeating instructions.

It even double-checks itself before… https://t.co/ssuJZoDjUn pic.twitter.com/eNsFqt9VfS
— vladuah (@vladuah) July 4, 2026

@vladuah's July 4 thread on Hermes Agent's GUI workflow capture hit 22 likes, 13 bookmarks, and 5,171 impressions. The underlying feature is open PR #50138 (by @Snail3D, opened 2026-06-21): "feat(skills): Record and Replay - record macOS workflows via CGEventTap, generate replayable skills." It is 8 files and +4,372 lines of code, all in optional-skills/automation/record-and-replay/.

The PR description frames the inspiration this way: a port of OpenAI Codex's Record & Replay plugin, with a "Cadillac twist" - real input event capture via CGEventTap, not screenshot diffing.

The three-phase pipeline

The skill ships three scripts that run sequentially:

1. Record - record_workflow.py (1,140 LOC) attaches a listen-only CGEventTap to capture mouse clicks (button, position, modifiers), keystrokes (keycode, character, modifiers), scrolls, drags, and frontmost-app changes. Periodically it dumps screenshots via screencapture and accessibility (AX) trees via cua-driver (with osascript fallback). The full recording lands at:

~/.hermes/recordings/<timestamp>/
+-- metadata.json
+-- events/events.jsonl
+-- screenshots/0001.png ...
`-- ax_trees/0001.txt ...

2. Generate - analyze_recording.py (789 LOC) groups raw events into logical steps using sliding-window detection: a 1.5-second pause, an app switch, a pixel-changed screenshot, or a clipboard copy each marks a step boundary. It detects retries (nearby clicks within 500 ms) and patterns (repeated action-type signatures). The result is a structured JSON summary.

3. Replay - replay_skill.py (481 LOC) parses the generated SKILL.md, then for each step captures a fresh screenshot, identifies numbered UI elements, clicks by element index (not pixel coordinates, so window movement does not break replay), verifies state, and reports completion.

The agent is the glue. After analyze_recording.py writes the JSON, the agent uses vision_analyze to read the screenshots and skill_manage(action="create") to author the final SKILL.md with element descriptions, verification criteria, and error-recovery steps.

What's in v3.0.0 - production hardening

The initial PR body shows the bare pipeline. A PR comment on June 21 (also by Snail3D, also dated 2026-06-21) adds the production layer:

Screenshot diffing - PIL.ImageChops.difference with a 5% threshold. Only saves screenshots when pixels change. The comment reports "80%+ reduction in disk usage for typical workflows where the user pauses between actions."
Clipboard capture - polls pbpaste (macOS), xclip (Linux), or tkinter (Windows) every 0.5s. Logs the first 200 chars of each copy event. Captures paste workflows that pure key logging misses.
Window state tracking - NSWorkspace plus AX API logs window_resized, window_moved, window_minimized for replay accuracy across layout changes.
Retry detection in the analyzer - nearby clicks within 500 ms get flagged as mis-clicks and counted in the efficiency score.
HTML viewer - view_recording.html.template produces a self-contained file (inline CSS, base64 images, no external deps) with a play button at 1x, 2x, 5x, 10x speed.
Vision-based generation in generate_skill.py - calls the configured vision provider to describe UI elements. Falls back to text-only (AX tree plus event log) when no vision provider is configured.

The v3.0.0 comment lists 7 files, +2,366/-325 LOC; the v4.0.0 self-learning module adds a 631-line eighth file.

v4.0.0 - the agent records itself

The headline technical claim is in the second PR comment (v4.0.0, June 21). The agent now records its own computer_use actions and optimizes across attempts:

Agent starts recording in the background
Agent performs the task via computer_use (attempt 1 - slow, with retries)
self_learn.py --analyze computes an efficiency score per attempt
Agent tries again (attempt 2 - faster, fewer mistakes)
Agent optimizes (attempt 3 - shortcuts, combined steps)
self_learn.py --compare ranks all attempts
self_learn.py --generate authors a SKILL.md from the best attempt

The efficiency score is a weighted sum, lower-is-better:

Metric	Weight	What it measures
Action count	30%	Total clicks + keys + scrolls
Retry count	30%	Detected mis-clicks within 500 ms windows
Duration	20%	Wall time for the attempt
Step count	20%	Logical steps detected by the analyzer

The PR comment reports a synthetic test case ("3 attempts with decreasing inefficiency") with scores 3.01 -> 1.46, a 51.5% reduction. A worked Bambu Studio example in the same comment shows the per-attempt trajectory:

Attempt	Score	What changed
1 - explore	7.34	Agent clicks wrong menus, retries
2 - know the layout	4.12	Fewer clicks, fewer retries
3 - optimized	2.56	Keyboard shortcuts, combined steps
Reduction a1 to a3	-65.1%	Score 7.34 to 2.56

The best attempt becomes the replayable skill. Future invocations of the same task replay attempt 3's path.

File-level breakdown

File	+LOC	Role
record_workflow.py	1,140	CGEventTap capture, screenshots, AX trees
analyze_recording.py	789	Event grouping, pattern and retry detection
self_learn.py	631	Multi-attempt optimization, scoring, generation
SKILL.md	515	Full skill documentation
replay_skill.py	481	Capture, find, act, verify, retry engine
generate_skill.py	432	Vision-based SKILL.md authoring
test_record_replay.sh	264	Integration test (synthetic record to replay)
view_recording.html.template	120	Self-contained HTML timeline viewer
Total	4,372	8 files, 0 deletions, 0 core changes

Placement matters

The PR lives at optional-skills/automation/record-and-replay/. It is a skill, not a core tool - discoverable via hermes skills browse, installable with hermes skills install. No new HERMES_* env vars. No core changes. No prompt caching impact. Per the PR description, this is the recommended pattern from "Should it be a Skill or a Tool?" - the recorder is provider-agnostic and macOS-specific, two properties that argue against promoting it to a built-in tool.

The four external dependencies are all on macOS: pyobjc-framework-Quartz for CGEventTap, cua-driver for AX trees (with osascript fallback), screencapture for screenshots, and Accessibility plus Screen Recording permissions for the terminal app.

Earlier thread

Issue #19802, opened 2026-05-04, requested "GUI Teaching for Hermes Agent - Record mouse/keyboard workflows as executable skills (macOS)" and pointed at VodoooFilms/Linka, a third-party macOS remote-control app with Hermes integration that already shipped the pattern. PR #50138 is the in-repo implementation, with CGEventTap capture instead of the screenshot-diffing approach that prior implementations used.

The PR is open as of 2026-07-05. The synthetic self-learning test passes (3 attempts, best score 1.46 vs 3.01 baseline). MacOS 26.3 is the tested platform.

[^1]: vladuah. "Hermes Agent now learns to repeat your boring tasks after watching once." X. 2026-07-04. [^2]: Snail3D. "feat(skills): Record and Replay - record macOS workflows via CGEventTap, generate replayable skills." PR #50138. NousResearch/hermes-agent. Opened 2026-06-21. [^3]: Snail3D. "Production-grade improvements - v3.0.0" (PR comment). 2026-06-21. [^4]: Snail3D. "Self-Learning Mode Added (v4.0.0)" (PR comment). 2026-06-21. [^5]: "GUI Teaching for Hermes Agent - Record mouse/keyboard workflows as executable skills (macOS)." Issue #19802. NousResearch/hermes-agent. Opened 2026-05-04. [^6]: HodlReaper. "An open-source agent cut its own runtime from 102 seconds to 35. Nobody touched the code." X. 2026-07-04. [^7]: interowhy. "One creator automated repetitive tasks with Hermes Agent and saved 15 hours per week." X. 2026-07-05.