docs(hermes_infra): add gateway-lag-recovery runbook #30

Open
hermes wants to merge 1 commits from hermes/ansible:docs/hermes_infra-gateway-lag-recovery into master
Contributor

Summary

Add an operator-facing runbook for the warning → drop cascade that produced admin/ansible#26. The file lives at hermes_squid/diagnostics/gateway-lag-recovery.md (4 KB, single doc).

Why

admin/ansible#26 traced a user-visible "session interrupted" symptom to a specific chain:

Persisted transcript lagged … (possible FTS write corruption)
  → cgroup memory pressure (1G limit)
  → s6/SIGTERM (no healthcheck → swarm cannot tell "alive but slow to disk" from "dead")
  → drain timed out after 0.0s (default 10s stop_grace_period is too short)
  → interrupting remaining work
  → Scheduled auto-resume

Until PRs #27 (healthcheck + stop_grace_period) and #28 (memory 1G→2G) ship and roll out to prod, anyone on call will see this warning and need a known-good mitigation ladder. Today there is no runbook — the only context is the issue body.

Diff

  • hermes_squid/diagnostics/gateway-lag-recovery.md (new, 74 lines).
  • hermes_squid/diagnostics/ directory created.

No compose change. No deploy step. Pure documentation.

Test plan

  1. git ls-files hermes_squid/diagnostics/ lists gateway-lag-recovery.md.
  2. Open the file on gitea — renders as a usable page.
  3. Verify the diagnostic commands work on a live prod container:
    CID=$(docker ps -qf name=hermes_infra_hermes-gateway)
    docker exec "$CID" sh -c 'cat /proc/$(pgrep -f "hermes gateway run")/status' | grep -E '^Vm' | head
    
    This should print VmRSS, VmHWM, etc. without error.

Rollback

Documentation only — git revert HEAD or delete the directory.

Refs: admin/ansible#26.

## Summary Add an operator-facing runbook for the warning → drop cascade that produced `admin/ansible#26`. The file lives at `hermes_squid/diagnostics/gateway-lag-recovery.md` (4 KB, single doc). ## Why `admin/ansible#26` traced a user-visible "session interrupted" symptom to a specific chain: ``` Persisted transcript lagged … (possible FTS write corruption) → cgroup memory pressure (1G limit) → s6/SIGTERM (no healthcheck → swarm cannot tell "alive but slow to disk" from "dead") → drain timed out after 0.0s (default 10s stop_grace_period is too short) → interrupting remaining work → Scheduled auto-resume ``` Until PRs #27 (healthcheck + stop_grace_period) and #28 (memory 1G→2G) ship and roll out to prod, anyone on call will see this warning and need a known-good mitigation ladder. Today there is no runbook — the only context is the issue body. ## Diff - `hermes_squid/diagnostics/gateway-lag-recovery.md` (new, 74 lines). - `hermes_squid/diagnostics/` directory created. No compose change. No deploy step. Pure documentation. ## Test plan 1. `git ls-files hermes_squid/diagnostics/` lists `gateway-lag-recovery.md`. 2. Open the file on gitea — renders as a usable page. 3. Verify the diagnostic commands work on a live prod container: ```bash CID=$(docker ps -qf name=hermes_infra_hermes-gateway) docker exec "$CID" sh -c 'cat /proc/$(pgrep -f "hermes gateway run")/status' | grep -E '^Vm' | head ``` This should print `VmRSS`, `VmHWM`, etc. without error. ## Rollback Documentation only — `git revert HEAD` or delete the directory. Refs: `admin/ansible#26`.
hermes added 1 commit 2026-08-30 23:43:32 +02:00
On-call playbook for the symptom chain that produced the prod outage
on admin/ansible#26:

  Persisted transcript lagged
    -> cgroup memory pressure
    -> s6 SIGTERM
    -> drain timed out 0.0s
    -> interrupting remaining work
    -> Scheduled auto-resume

Includes live diagnostic commands (gateway log grep, cgroup RSS /
threads inside the container, WAL snapshot), a four-step mitigation
ladder (clear-session, USR2 checkpoint, rolling restart, escalate)
with PR #27 / PR #28 cross-refs, and an explicit 'what NOT to do'
list (docker kill without run id, log rotate mid-checkpoint, masked
timeouts).

Refs admin/ansible#26.
Checking for merge conflicts…
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u https://gitea.rudolf.keenetic.link/hermes/ansible docs/hermes_infra-gateway-lag-recovery:hermes-docs/hermes_infra-gateway-lag-recovery
git checkout hermes-docs/hermes_infra-gateway-lag-recovery
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/ansible#30