From d926a622bef87da61a0c199c0b85093c8f1b7236 Mon Sep 17 00:00:00 2001 From: hermes-agent Date: Sun, 30 Aug 2026 21:43:13 +0000 Subject: [PATCH] docs(hermes_infra): add gateway-lag-recovery runbook On-call playbook for the symptom chain that produced the prod outage on admin/ansible#26: Persisted transcript lagged -> cgroup memory pressure -> s6 SIGTERM -> drain timed out 0.0s -> interrupting remaining work -> Scheduled auto-resume Includes live diagnostic commands (gateway log grep, cgroup RSS / threads inside the container, WAL snapshot), a four-step mitigation ladder (clear-session, USR2 checkpoint, rolling restart, escalate) with PR #27 / PR #28 cross-refs, and an explicit 'what NOT to do' list (docker kill without run id, log rotate mid-checkpoint, masked timeouts). Refs admin/ansible#26. --- .../diagnostics/gateway-lag-recovery.md | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 hermes_squid/diagnostics/gateway-lag-recovery.md diff --git a/hermes_squid/diagnostics/gateway-lag-recovery.md b/hermes_squid/diagnostics/gateway-lag-recovery.md new file mode 100644 index 0000000..24dc19f --- /dev/null +++ b/hermes_squid/diagnostics/gateway-lag-recovery.md @@ -0,0 +1,74 @@ +# Recovery: "Persisted transcript lagged (possible FTS write corruption)" + +Use this runbook when `hermes_infra_hermes-gateway` starts emitting the warning `WARNING gateway.run: Persisted transcript lagged live cached history for session (disk=N, memory=N+1); preserving live conversation context (possible FTS write corruption)` more than a handful of times per hour. + +## What it means + +Hermes writes session transcript entries to an in-memory ring and then flushes them to a SQLite WAL on disk. The warning fires when `disk` is more than one entry behind `memory` — i.e. flushes are too slow for the workload. It is **not** data loss by itself (memory still has the entries), but it is the upstream symptom of cascading failures: + +``` +Persisted transcript lagged → RSS climbs → cgroup memory pressure → + s6/SIGTERM → drain timed out 0.0s → interrupting remaining work → + Scheduled auto-resume → user-visible drop +``` + +Until PRs #27 (healthcheck+stop_grace_period) and #28 (memory 1G→2G) are deployed, expect cycles of this warning followed within minutes by a full service restart. + +## Quick check (run on the swarm manager) + +```bash +# 1. Tail the gateway log for the last warning count and the next SIGTERM. +CID=$(docker ps -qf name=hermes_infra_hermes-gateway) +docker exec "$CID" sh -c 'tail -n 200 /opt/data/logs/gateway.log' \ + | grep -E 'Persisted transcript lagged|SIGTERM|drain timed out|interrupting remaining work|Scheduled auto-resume' + +# 2. Memory + Pids inside the container today (live numbers). +docker exec "$CID" sh -c 'cat /proc/$(pgrep -f "hermes gateway run")/status' \ + | grep -E '^Vm(HWM|RSS|Peak|Size):|^Threads:' + +# 3. SQLite WAL stuck? Pending writes give a hint. +docker exec "$CID" sh -c 'ls -la /opt/data/sessions/ | head' \ + | head +``` + +A healthy gateway shows: + +- `VmRSS` under 1.5 GiB (gates on 1G before PR #28 lands). +- `Threads` under 200. +- No `(possible FTS write corruption)` warnings newer than ~5 minutes. +- No new `Received SIGTERM … parent_name=s6-supervise`. + +## Mitigation ladder + +Do **one** step, then re-check (1) and (2). Do not skip ahead. + +1. **Housekeeping: clear the largest sessions.** Long Home/thread-114 sessions are the dominant contributor. + ```bash + # find the largest sessions on disk + docker exec "$CID" sh -c \ + 'du -sh /opt/data/sessions//* | sort -h | tail -20' + # then `/clear` for the offenders in chat; Hermes rebuilds the in-memory copy on the next message. + ``` +2. **Force a checkpoint flush.** This drops the WAL to disk, resets the ring, and does not lose user-visible history. + ```bash + docker exec "$CID" sh -c \ + 'kill -USR2 $(pgrep -f "hermes gateway run")' # USR2 = checkpoint in Hermes CLI; verify via `docker exec ... cat /opt/data/gateway_state.json` + ``` +3. **Rolling restart of the service.** Use brencher if it is configured to redeploy `hermes_infra`; otherwise: + ```bash + docker service update --force --update-parallelism 1 --update-delay 10s hermes_infra_hermes-gateway + ``` + After this run PR #28 (memory 2G) in brencher first; otherwise the warning will return within an hour. +4. **If everything above fails, escalate.** Dump the gateway log + a `docker stats` snapshot, attach to `admin/ansible#26`. Do not skip the dump. + +## What NOT to do + +- Do not `docker kill` the gateway without recording the active run id from the last log line; that run will not be auto-resumed. +- Do not rotate `gateway.log` while a session is mid-checkpoint; you lose the last few writes that are still in memory. +- Do not increase `HEALTHCHECK.TimeoutSeconds` above 5s on TCP probes — that masks real daemon hangs and lets them stay unhealthy forever. + +## Sources + +- Trigger and incident timeline: `admin/ansible#26`. +- Pre-emptive fix on the way: PR #27 (healthcheck + stop_grace_period), PR #28 (memory 1G → 2G). +- Hermes CLI USR2 semantics: pulled from `/opt/hermes/hermes_cli/gateway.py:checkpoint` inside the container (no public docs; reverify after each Hermes image bump). -- 2.54.0