fix(hermes_infra): hermes-gateway healthcheck + stop_grace_period #27

Open
hermes wants to merge 1 commits from hermes/ansible:fix/hermes-gateway-healthcheck into master
Contributor

Summary

  • Add TCP healthcheck to hermes-gateway (stack hermes_infra) on the dashboard port 8087.
  • Add stop_grace_period: 60s to give Hermes time for a clean shutdown drain (default 10s leads to drain timed out after 0.0s and interrupting remaining work).
  • Tighten restart_policy to delay: 5s, max_attempts: 5 so the service flaps a bounded number of times instead of looping.

Why

admin/ansible#26 shows that the prod gateway receives SIGTERM from s6-supervise (parent_pid=148 parent_name=s6-supervise) when a long Home-session sits on a backlogged FTS write — the swarm daemon cannot tell "process is alive but slow to disk" from "process is dead", because the service has no healthcheck and the default 10s stop_grace_period is shorter than Hermes' drain. Two observed patterns today:

  • 18:45:38 … Received SIGTERM … parent_pid=148 parent_name=s6-supervise → drain timed out after 0.0s with 1 active agent(s) → interrupting remaining work → user-visible drop.
  • 22:53 — same root via a service Spec.Update from brencher (Spec.UpdatedAt=2026-08-29T22:53:16Z).

Add a passive liveness check that always succeeds when the gateway is up, but fails immediately on a TCP RST/timeout when the process is gone — and stretch the stop deadline so a healthy drain can complete.

Diff scope

Single file: hermes_squid/stack-compose.yml, +9 lines.

restart_policy:
  condition: on-failure
+ delay: 5s
+ max_attempts: 5
healthcheck:
  test: ["CMD", "python3", "-c", "import socket; s=socket.socket(); s.settimeout(2); s.connect(('127.0.0.1', 8087))"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 60s
stop_grace_period: 60s

Notes / caveats

  • Port 8087 is the gateway dashboard bind (HERMES_DASHBOARD_HOST=0.0.0.0); a successful TCP connect means the gateway event loop is alive. If a future image exposes a dedicated /health HTTP endpoint, swap the test for ["CMD", "wget", "-qO-", "http://127.0.0.1:PORT/health"].
  • stop_grace_period lives in services.<name>.stop_grace_period for compose-spec v3.9; brencher's docker stack deploy honours it. Source: Docker docs.
  • Memory / pids limits are intentionally not changed here — they live in PR #2.

Test plan (on prod)

  1. Merge this PR.
  2. From brencher (or docker service update --with-registry-image): redeploy hermes_infra only.
  3. docker service inspect hermes_infra_hermes-gateway | jq '.Spec.TaskTemplate.ContainerSpec.Healthcheck' → present.
  4. docker service ps hermes_infra_hermes-gateway → task state should report (healthy).
  5. Tail /opt/data/logs/gateway.log (hermes_infra_hermes-gateway) for 24 h: the 30s interval healthchecks should leave no traces (passive check). A drain timed out after 0.0s line should not appear.

Rollback

# brencher can `docker service rollback hermes_infra_hermes-gateway`
# or, with compose:
git revert -m 1 HEAD

Refs: admin/ansible#26.

## Summary - Add TCP healthcheck to `hermes-gateway` (stack `hermes_infra`) on the dashboard port `8087`. - Add `stop_grace_period: 60s` to give Hermes time for a clean shutdown drain (default 10s leads to `drain timed out after 0.0s` and `interrupting remaining work`). - Tighten `restart_policy` to `delay: 5s, max_attempts: 5` so the service flaps a bounded number of times instead of looping. ## Why `admin/ansible#26` shows that the prod gateway receives `SIGTERM` from `s6-supervise` (`parent_pid=148 parent_name=s6-supervise`) when a long Home-session sits on a backlogged FTS write — the swarm daemon cannot tell "process is alive but slow to disk" from "process is dead", because the service has **no healthcheck** and the default 10s `stop_grace_period` is shorter than Hermes' drain. Two observed patterns today: - `18:45:38 … Received SIGTERM … parent_pid=148 parent_name=s6-supervise` → `drain timed out after 0.0s with 1 active agent(s)` → `interrupting remaining work` → user-visible drop. - `22:53` — same root via a `service Spec.Update` from brencher (`Spec.UpdatedAt=2026-08-29T22:53:16Z`). Add a passive liveness check that always succeeds when the gateway is up, but **fails immediately on a TCP RST/timeout when the process is gone** — and stretch the stop deadline so a healthy drain can complete. ## Diff scope Single file: `hermes_squid/stack-compose.yml`, +9 lines. ```yaml restart_policy: condition: on-failure + delay: 5s + max_attempts: 5 healthcheck: test: ["CMD", "python3", "-c", "import socket; s=socket.socket(); s.settimeout(2); s.connect(('127.0.0.1', 8087))"] interval: 30s timeout: 5s retries: 3 start_period: 60s stop_grace_period: 60s ``` ## Notes / caveats - Port `8087` is the gateway dashboard bind (`HERMES_DASHBOARD_HOST=0.0.0.0`); a successful TCP connect means the gateway event loop is alive. If a future image exposes a dedicated `/health` HTTP endpoint, swap the test for `["CMD", "wget", "-qO-", "http://127.0.0.1:PORT/health"]`. - `stop_grace_period` lives in `services.<name>.stop_grace_period` for `compose-spec` v3.9; brencher's `docker stack deploy` honours it. Source: Docker docs. - Memory / pids limits are intentionally **not** changed here — they live in PR #2. ## Test plan (on prod) 1. Merge this PR. 2. From brencher (or `docker service update --with-registry-image`): redeploy `hermes_infra` only. 3. `docker service inspect hermes_infra_hermes-gateway | jq '.Spec.TaskTemplate.ContainerSpec.Healthcheck'` → present. 4. `docker service ps hermes_infra_hermes-gateway` → task state should report `(healthy)`. 5. Tail `/opt/data/logs/gateway.log` (`hermes_infra_hermes-gateway`) for 24 h: the 30s `interval` healthchecks should leave no traces (passive check). A `drain timed out after 0.0s` line should not appear. ## Rollback ```bash # brencher can `docker service rollback hermes_infra_hermes-gateway` # or, with compose: git revert -m 1 HEAD ``` Refs: `admin/ansible#26`.
hermes added 1 commit 2026-08-30 23:41:11 +02:00
prod-gateway gets SIGTERM from s6 because no healthcheck exists and
default stop_grace_period (10s) is shorter than Hermes' shutdown drain.
Add TCP healthcheck on the dashboard port (8087), stop_grace_period=60s,
and a backoff (delay=5s, max_attempts=5) on restart_policy so the daemon
survives a long FTS backlog without flapping into 'interrupting remaining
work' / 'Scheduled auto-resume' cycles.

Refs admin/ansible#26.
Checking for merge conflicts…
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u https://gitea.rudolf.keenetic.link/hermes/ansible fix/hermes-gateway-healthcheck:hermes-fix/hermes-gateway-healthcheck
git checkout hermes-fix/hermes-gateway-healthcheck
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/ansible#27