Add an operator-facing runbook for the warning → drop cascade that produced admin/ansible#26. The file lives at hermes_squid/diagnostics/gateway-lag-recovery.md (4 KB, single doc).
Why
admin/ansible#26 traced a user-visible "session interrupted" symptom to a specific chain:
Persisted transcript lagged … (possible FTS write corruption)
→ cgroup memory pressure (1G limit)
→ s6/SIGTERM (no healthcheck → swarm cannot tell "alive but slow to disk" from "dead")
→ drain timed out after 0.0s (default 10s stop_grace_period is too short)
→ interrupting remaining work
→ Scheduled auto-resume
Until PRs #27 (healthcheck + stop_grace_period) and #28 (memory 1G→2G) ship and roll out to prod, anyone on call will see this warning and need a known-good mitigation ladder. Today there is no runbook — the only context is the issue body.
Open the file on gitea — renders as a usable page.
Verify the diagnostic commands work on a live prod container:
CID=$(docker ps -qf name=hermes_infra_hermes-gateway)
docker exec"$CID" sh -c 'cat /proc/$(pgrep -f "hermes gateway run")/status'| grep -E '^Vm'| head
This should print VmRSS, VmHWM, etc. without error.
Rollback
Documentation only — git revert HEAD or delete the directory.
Refs: admin/ansible#26.
## Summary
Add an operator-facing runbook for the warning → drop cascade that produced `admin/ansible#26`. The file lives at `hermes_squid/diagnostics/gateway-lag-recovery.md` (4 KB, single doc).
## Why
`admin/ansible#26` traced a user-visible "session interrupted" symptom to a specific chain:
```
Persisted transcript lagged … (possible FTS write corruption)
→ cgroup memory pressure (1G limit)
→ s6/SIGTERM (no healthcheck → swarm cannot tell "alive but slow to disk" from "dead")
→ drain timed out after 0.0s (default 10s stop_grace_period is too short)
→ interrupting remaining work
→ Scheduled auto-resume
```
Until PRs #27 (healthcheck + stop_grace_period) and #28 (memory 1G→2G) ship and roll out to prod, anyone on call will see this warning and need a known-good mitigation ladder. Today there is no runbook — the only context is the issue body.
## Diff
- `hermes_squid/diagnostics/gateway-lag-recovery.md` (new, 74 lines).
- `hermes_squid/diagnostics/` directory created.
No compose change. No deploy step. Pure documentation.
## Test plan
1. `git ls-files hermes_squid/diagnostics/` lists `gateway-lag-recovery.md`.
2. Open the file on gitea — renders as a usable page.
3. Verify the diagnostic commands work on a live prod container:
```bash
CID=$(docker ps -qf name=hermes_infra_hermes-gateway)
docker exec "$CID" sh -c 'cat /proc/$(pgrep -f "hermes gateway run")/status' | grep -E '^Vm' | head
```
This should print `VmRSS`, `VmHWM`, etc. without error.
## Rollback
Documentation only — `git revert HEAD` or delete the directory.
Refs: `admin/ansible#26`.
On-call playbook for the symptom chain that produced the prod outage
on admin/ansible#26:
Persisted transcript lagged
-> cgroup memory pressure
-> s6 SIGTERM
-> drain timed out 0.0s
-> interrupting remaining work
-> Scheduled auto-resume
Includes live diagnostic commands (gateway log grep, cgroup RSS /
threads inside the container, WAL snapshot), a four-step mitigation
ladder (clear-session, USR2 checkpoint, rolling restart, escalate)
with PR #27 / PR #28 cross-refs, and an explicit 'what NOT to do'
list (docker kill without run id, log rotate mid-checkpoint, masked
timeouts).
Refs admin/ansible#26.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Add an operator-facing runbook for the warning → drop cascade that produced
admin/ansible#26. The file lives athermes_squid/diagnostics/gateway-lag-recovery.md(4 KB, single doc).Why
admin/ansible#26traced a user-visible "session interrupted" symptom to a specific chain:Until PRs #27 (healthcheck + stop_grace_period) and #28 (memory 1G→2G) ship and roll out to prod, anyone on call will see this warning and need a known-good mitigation ladder. Today there is no runbook — the only context is the issue body.
Diff
hermes_squid/diagnostics/gateway-lag-recovery.md(new, 74 lines).hermes_squid/diagnostics/directory created.No compose change. No deploy step. Pure documentation.
Test plan
git ls-files hermes_squid/diagnostics/listsgateway-lag-recovery.md.VmRSS,VmHWM, etc. without error.Rollback
Documentation only —
git revert HEADor delete the directory.Refs:
admin/ansible#26.On-call playbook for the symptom chain that produced the prod outage on admin/ansible#26: Persisted transcript lagged -> cgroup memory pressure -> s6 SIGTERM -> drain timed out 0.0s -> interrupting remaining work -> Scheduled auto-resume Includes live diagnostic commands (gateway log grep, cgroup RSS / threads inside the container, WAL snapshot), a four-step mitigation ladder (clear-session, USR2 checkpoint, rolling restart, escalate) with PR #27 / PR #28 cross-refs, and an explicit 'what NOT to do' list (docker kill without run id, log rotate mid-checkpoint, masked timeouts). Refs admin/ansible#26.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.