docs(hermes_infra): add gateway-lag-recovery runbook
On-call playbook for the symptom chain that produced the prod outage
on admin/ansible#26:
Persisted transcript lagged
-> cgroup memory pressure
-> s6 SIGTERM
-> drain timed out 0.0s
-> interrupting remaining work
-> Scheduled auto-resume
Includes live diagnostic commands (gateway log grep, cgroup RSS /
threads inside the container, WAL snapshot), a four-step mitigation
ladder (clear-session, USR2 checkpoint, rolling restart, escalate)
with PR #27 / PR #28 cross-refs, and an explicit 'what NOT to do'
list (docker kill without run id, log rotate mid-checkpoint, masked
timeouts).
Refs admin/ansible#26.
This commit is contained in:
@@ -0,0 +1,74 @@
|
||||
# Recovery: "Persisted transcript lagged (possible FTS write corruption)"
|
||||
|
||||
Use this runbook when `hermes_infra_hermes-gateway` starts emitting the warning `WARNING gateway.run: Persisted transcript lagged live cached history for session <sid> (disk=N, memory=N+1); preserving live conversation context (possible FTS write corruption)` more than a handful of times per hour.
|
||||
|
||||
## What it means
|
||||
|
||||
Hermes writes session transcript entries to an in-memory ring and then flushes them to a SQLite WAL on disk. The warning fires when `disk` is more than one entry behind `memory` — i.e. flushes are too slow for the workload. It is **not** data loss by itself (memory still has the entries), but it is the upstream symptom of cascading failures:
|
||||
|
||||
```
|
||||
Persisted transcript lagged → RSS climbs → cgroup memory pressure →
|
||||
s6/SIGTERM → drain timed out 0.0s → interrupting remaining work →
|
||||
Scheduled auto-resume → user-visible drop
|
||||
```
|
||||
|
||||
Until PRs #27 (healthcheck+stop_grace_period) and #28 (memory 1G→2G) are deployed, expect cycles of this warning followed within minutes by a full service restart.
|
||||
|
||||
## Quick check (run on the swarm manager)
|
||||
|
||||
```bash
|
||||
# 1. Tail the gateway log for the last warning count and the next SIGTERM.
|
||||
CID=$(docker ps -qf name=hermes_infra_hermes-gateway)
|
||||
docker exec "$CID" sh -c 'tail -n 200 /opt/data/logs/gateway.log' \
|
||||
| grep -E 'Persisted transcript lagged|SIGTERM|drain timed out|interrupting remaining work|Scheduled auto-resume'
|
||||
|
||||
# 2. Memory + Pids inside the container today (live numbers).
|
||||
docker exec "$CID" sh -c 'cat /proc/$(pgrep -f "hermes gateway run")/status' \
|
||||
| grep -E '^Vm(HWM|RSS|Peak|Size):|^Threads:'
|
||||
|
||||
# 3. SQLite WAL stuck? Pending writes give a hint.
|
||||
docker exec "$CID" sh -c 'ls -la /opt/data/sessions/ | head' \
|
||||
| head
|
||||
```
|
||||
|
||||
A healthy gateway shows:
|
||||
|
||||
- `VmRSS` under 1.5 GiB (gates on 1G before PR #28 lands).
|
||||
- `Threads` under 200.
|
||||
- No `(possible FTS write corruption)` warnings newer than ~5 minutes.
|
||||
- No new `Received SIGTERM … parent_name=s6-supervise`.
|
||||
|
||||
## Mitigation ladder
|
||||
|
||||
Do **one** step, then re-check (1) and (2). Do not skip ahead.
|
||||
|
||||
1. **Housekeeping: clear the largest sessions.** Long Home/thread-114 sessions are the dominant contributor.
|
||||
```bash
|
||||
# find the largest sessions on disk
|
||||
docker exec "$CID" sh -c \
|
||||
'du -sh /opt/data/sessions/<agent:main:telegram:group:-1004417969359:*>/* | sort -h | tail -20'
|
||||
# then `/clear` for the offenders in chat; Hermes rebuilds the in-memory copy on the next message.
|
||||
```
|
||||
2. **Force a checkpoint flush.** This drops the WAL to disk, resets the ring, and does not lose user-visible history.
|
||||
```bash
|
||||
docker exec "$CID" sh -c \
|
||||
'kill -USR2 $(pgrep -f "hermes gateway run")' # USR2 = checkpoint in Hermes CLI; verify via `docker exec ... cat /opt/data/gateway_state.json`
|
||||
```
|
||||
3. **Rolling restart of the service.** Use brencher if it is configured to redeploy `hermes_infra`; otherwise:
|
||||
```bash
|
||||
docker service update --force --update-parallelism 1 --update-delay 10s hermes_infra_hermes-gateway
|
||||
```
|
||||
After this run PR #28 (memory 2G) in brencher first; otherwise the warning will return within an hour.
|
||||
4. **If everything above fails, escalate.** Dump the gateway log + a `docker stats` snapshot, attach to `admin/ansible#26`. Do not skip the dump.
|
||||
|
||||
## What NOT to do
|
||||
|
||||
- Do not `docker kill` the gateway without recording the active run id from the last log line; that run will not be auto-resumed.
|
||||
- Do not rotate `gateway.log` while a session is mid-checkpoint; you lose the last few writes that are still in memory.
|
||||
- Do not increase `HEALTHCHECK.TimeoutSeconds` above 5s on TCP probes — that masks real daemon hangs and lets them stay unhealthy forever.
|
||||
|
||||
## Sources
|
||||
|
||||
- Trigger and incident timeline: `admin/ansible#26`.
|
||||
- Pre-emptive fix on the way: PR #27 (healthcheck + stop_grace_period), PR #28 (memory 1G → 2G).
|
||||
- Hermes CLI USR2 semantics: pulled from `/opt/hermes/hermes_cli/gateway.py:checkpoint` inside the container (no public docs; reverify after each Hermes image bump).
|
||||
Reference in New Issue
Block a user