docs(hermes_infra): add gateway-lag-recovery runbook

On-call playbook for the symptom chain that produced the prod outage
on admin/ansible#26:

  Persisted transcript lagged
    -> cgroup memory pressure
    -> s6 SIGTERM
    -> drain timed out 0.0s
    -> interrupting remaining work
    -> Scheduled auto-resume

Includes live diagnostic commands (gateway log grep, cgroup RSS /
threads inside the container, WAL snapshot), a four-step mitigation
ladder (clear-session, USR2 checkpoint, rolling restart, escalate)
with PR #27 / PR #28 cross-refs, and an explicit 'what NOT to do'
list (docker kill without run id, log rotate mid-checkpoint, masked
timeouts).

Refs admin/ansible#26.
This commit is contained in:
2026-08-30 21:43:13 +00:00
parent c3b7128006
commit d926a622be
@@ -0,0 +1,74 @@
# Recovery: "Persisted transcript lagged (possible FTS write corruption)"
Use this runbook when `hermes_infra_hermes-gateway` starts emitting the warning `WARNING gateway.run: Persisted transcript lagged live cached history for session <sid> (disk=N, memory=N+1); preserving live conversation context (possible FTS write corruption)` more than a handful of times per hour.
## What it means
Hermes writes session transcript entries to an in-memory ring and then flushes them to a SQLite WAL on disk. The warning fires when `disk` is more than one entry behind `memory` — i.e. flushes are too slow for the workload. It is **not** data loss by itself (memory still has the entries), but it is the upstream symptom of cascading failures:
```
Persisted transcript lagged → RSS climbs → cgroup memory pressure →
s6/SIGTERM → drain timed out 0.0s → interrupting remaining work →
Scheduled auto-resume → user-visible drop
```
Until PRs #27 (healthcheck+stop_grace_period) and #28 (memory 1G→2G) are deployed, expect cycles of this warning followed within minutes by a full service restart.
## Quick check (run on the swarm manager)
```bash
# 1. Tail the gateway log for the last warning count and the next SIGTERM.
CID=$(docker ps -qf name=hermes_infra_hermes-gateway)
docker exec "$CID" sh -c 'tail -n 200 /opt/data/logs/gateway.log' \
| grep -E 'Persisted transcript lagged|SIGTERM|drain timed out|interrupting remaining work|Scheduled auto-resume'
# 2. Memory + Pids inside the container today (live numbers).
docker exec "$CID" sh -c 'cat /proc/$(pgrep -f "hermes gateway run")/status' \
| grep -E '^Vm(HWM|RSS|Peak|Size):|^Threads:'
# 3. SQLite WAL stuck? Pending writes give a hint.
docker exec "$CID" sh -c 'ls -la /opt/data/sessions/ | head' \
| head
```
A healthy gateway shows:
- `VmRSS` under 1.5 GiB (gates on 1G before PR #28 lands).
- `Threads` under 200.
- No `(possible FTS write corruption)` warnings newer than ~5 minutes.
- No new `Received SIGTERM … parent_name=s6-supervise`.
## Mitigation ladder
Do **one** step, then re-check (1) and (2). Do not skip ahead.
1. **Housekeeping: clear the largest sessions.** Long Home/thread-114 sessions are the dominant contributor.
```bash
# find the largest sessions on disk
docker exec "$CID" sh -c \
'du -sh /opt/data/sessions/<agent:main:telegram:group:-1004417969359:*>/* | sort -h | tail -20'
# then `/clear` for the offenders in chat; Hermes rebuilds the in-memory copy on the next message.
```
2. **Force a checkpoint flush.** This drops the WAL to disk, resets the ring, and does not lose user-visible history.
```bash
docker exec "$CID" sh -c \
'kill -USR2 $(pgrep -f "hermes gateway run")' # USR2 = checkpoint in Hermes CLI; verify via `docker exec ... cat /opt/data/gateway_state.json`
```
3. **Rolling restart of the service.** Use brencher if it is configured to redeploy `hermes_infra`; otherwise:
```bash
docker service update --force --update-parallelism 1 --update-delay 10s hermes_infra_hermes-gateway
```
After this run PR #28 (memory 2G) in brencher first; otherwise the warning will return within an hour.
4. **If everything above fails, escalate.** Dump the gateway log + a `docker stats` snapshot, attach to `admin/ansible#26`. Do not skip the dump.
## What NOT to do
- Do not `docker kill` the gateway without recording the active run id from the last log line; that run will not be auto-resumed.
- Do not rotate `gateway.log` while a session is mid-checkpoint; you lose the last few writes that are still in memory.
- Do not increase `HEALTHCHECK.TimeoutSeconds` above 5s on TCP probes — that masks real daemon hangs and lets them stay unhealthy forever.
## Sources
- Trigger and incident timeline: `admin/ansible#26`.
- Pre-emptive fix on the way: PR #27 (healthcheck + stop_grace_period), PR #28 (memory 1G → 2G).
- Hermes CLI USR2 semantics: pulled from `/opt/hermes/hermes_cli/gateway.py:checkpoint` inside the container (no public docs; reverify after each Hermes image bump).