docs(hermes_infra): add gateway-lag-recovery runbook #30

Open
hermes wants to merge 1 commits from hermes/ansible:docs/hermes_infra-gateway-lag-recovery into master
1 Commits
Author SHA1 Message Date
hermes d926a622be docs(hermes_infra): add gateway-lag-recovery runbook
On-call playbook for the symptom chain that produced the prod outage
on admin/ansible#26:

  Persisted transcript lagged
    -> cgroup memory pressure
    -> s6 SIGTERM
    -> drain timed out 0.0s
    -> interrupting remaining work
    -> Scheduled auto-resume

Includes live diagnostic commands (gateway log grep, cgroup RSS /
threads inside the container, WAL snapshot), a four-step mitigation
ladder (clear-session, USR2 checkpoint, rolling restart, escalate)
with PR #27 / PR #28 cross-refs, and an explicit 'what NOT to do'
list (docker kill without run id, log rotate mid-checkpoint, masked
timeouts).

Refs admin/ansible#26.
2026-08-30 21:43:13 +00:00