Add TCP healthcheck to hermes-gateway (stack hermes_infra) on the dashboard port 8087.
Add stop_grace_period: 60s to give Hermes time for a clean shutdown drain (default 10s leads to drain timed out after 0.0s and interrupting remaining work).
Tighten restart_policy to delay: 5s, max_attempts: 5 so the service flaps a bounded number of times instead of looping.
Why
admin/ansible#26 shows that the prod gateway receives SIGTERM from s6-supervise (parent_pid=148 parent_name=s6-supervise) when a long Home-session sits on a backlogged FTS write — the swarm daemon cannot tell "process is alive but slow to disk" from "process is dead", because the service has no healthcheck and the default 10s stop_grace_period is shorter than Hermes' drain. Two observed patterns today:
18:45:38 … Received SIGTERM … parent_pid=148 parent_name=s6-supervise → drain timed out after 0.0s with 1 active agent(s) → interrupting remaining work → user-visible drop.
22:53 — same root via a service Spec.Update from brencher (Spec.UpdatedAt=2026-08-29T22:53:16Z).
Add a passive liveness check that always succeeds when the gateway is up, but fails immediately on a TCP RST/timeout when the process is gone — and stretch the stop deadline so a healthy drain can complete.
Diff scope
Single file: hermes_squid/stack-compose.yml, +9 lines.
Port 8087 is the gateway dashboard bind (HERMES_DASHBOARD_HOST=0.0.0.0); a successful TCP connect means the gateway event loop is alive. If a future image exposes a dedicated /health HTTP endpoint, swap the test for ["CMD", "wget", "-qO-", "http://127.0.0.1:PORT/health"].
stop_grace_period lives in services.<name>.stop_grace_period for compose-spec v3.9; brencher's docker stack deploy honours it. Source: Docker docs.
Memory / pids limits are intentionally not changed here — they live in PR #2.
Test plan (on prod)
Merge this PR.
From brencher (or docker service update --with-registry-image): redeploy hermes_infra only.
docker service inspect hermes_infra_hermes-gateway | jq '.Spec.TaskTemplate.ContainerSpec.Healthcheck' → present.
docker service ps hermes_infra_hermes-gateway → task state should report (healthy).
Tail /opt/data/logs/gateway.log (hermes_infra_hermes-gateway) for 24 h: the 30s interval healthchecks should leave no traces (passive check). A drain timed out after 0.0s line should not appear.
Rollback
# brencher can `docker service rollback hermes_infra_hermes-gateway`# or, with compose:
git revert -m 1 HEAD
Refs: admin/ansible#26.
## Summary
- Add TCP healthcheck to `hermes-gateway` (stack `hermes_infra`) on the dashboard port `8087`.
- Add `stop_grace_period: 60s` to give Hermes time for a clean shutdown drain (default 10s leads to `drain timed out after 0.0s` and `interrupting remaining work`).
- Tighten `restart_policy` to `delay: 5s, max_attempts: 5` so the service flaps a bounded number of times instead of looping.
## Why
`admin/ansible#26` shows that the prod gateway receives `SIGTERM` from `s6-supervise` (`parent_pid=148 parent_name=s6-supervise`) when a long Home-session sits on a backlogged FTS write — the swarm daemon cannot tell "process is alive but slow to disk" from "process is dead", because the service has **no healthcheck** and the default 10s `stop_grace_period` is shorter than Hermes' drain. Two observed patterns today:
- `18:45:38 … Received SIGTERM … parent_pid=148 parent_name=s6-supervise` → `drain timed out after 0.0s with 1 active agent(s)` → `interrupting remaining work` → user-visible drop.
- `22:53` — same root via a `service Spec.Update` from brencher (`Spec.UpdatedAt=2026-08-29T22:53:16Z`).
Add a passive liveness check that always succeeds when the gateway is up, but **fails immediately on a TCP RST/timeout when the process is gone** — and stretch the stop deadline so a healthy drain can complete.
## Diff scope
Single file: `hermes_squid/stack-compose.yml`, +9 lines.
```yaml
restart_policy:
condition: on-failure
+ delay: 5s
+ max_attempts: 5
healthcheck:
test: ["CMD", "python3", "-c", "import socket; s=socket.socket(); s.settimeout(2); s.connect(('127.0.0.1', 8087))"]
interval: 30s
timeout: 5s
retries: 3
start_period: 60s
stop_grace_period: 60s
```
## Notes / caveats
- Port `8087` is the gateway dashboard bind (`HERMES_DASHBOARD_HOST=0.0.0.0`); a successful TCP connect means the gateway event loop is alive. If a future image exposes a dedicated `/health` HTTP endpoint, swap the test for `["CMD", "wget", "-qO-", "http://127.0.0.1:PORT/health"]`.
- `stop_grace_period` lives in `services.<name>.stop_grace_period` for `compose-spec` v3.9; brencher's `docker stack deploy` honours it. Source: Docker docs.
- Memory / pids limits are intentionally **not** changed here — they live in PR #2.
## Test plan (on prod)
1. Merge this PR.
2. From brencher (or `docker service update --with-registry-image`): redeploy `hermes_infra` only.
3. `docker service inspect hermes_infra_hermes-gateway | jq '.Spec.TaskTemplate.ContainerSpec.Healthcheck'` → present.
4. `docker service ps hermes_infra_hermes-gateway` → task state should report `(healthy)`.
5. Tail `/opt/data/logs/gateway.log` (`hermes_infra_hermes-gateway`) for 24 h: the 30s `interval` healthchecks should leave no traces (passive check). A `drain timed out after 0.0s` line should not appear.
## Rollback
```bash
# brencher can `docker service rollback hermes_infra_hermes-gateway`
# or, with compose:
git revert -m 1 HEAD
```
Refs: `admin/ansible#26`.
prod-gateway gets SIGTERM from s6 because no healthcheck exists and
default stop_grace_period (10s) is shorter than Hermes' shutdown drain.
Add TCP healthcheck on the dashboard port (8087), stop_grace_period=60s,
and a backoff (delay=5s, max_attempts=5) on restart_policy so the daemon
survives a long FTS backlog without flapping into 'interrupting remaining
work' / 'Scheduled auto-resume' cycles.
Refs admin/ansible#26.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
hermes-gateway(stackhermes_infra) on the dashboard port8087.stop_grace_period: 60sto give Hermes time for a clean shutdown drain (default 10s leads todrain timed out after 0.0sandinterrupting remaining work).restart_policytodelay: 5s, max_attempts: 5so the service flaps a bounded number of times instead of looping.Why
admin/ansible#26shows that the prod gateway receivesSIGTERMfroms6-supervise(parent_pid=148 parent_name=s6-supervise) when a long Home-session sits on a backlogged FTS write — the swarm daemon cannot tell "process is alive but slow to disk" from "process is dead", because the service has no healthcheck and the default 10sstop_grace_periodis shorter than Hermes' drain. Two observed patterns today:18:45:38 … Received SIGTERM … parent_pid=148 parent_name=s6-supervise→drain timed out after 0.0s with 1 active agent(s)→interrupting remaining work→ user-visible drop.22:53— same root via aservice Spec.Updatefrom brencher (Spec.UpdatedAt=2026-08-29T22:53:16Z).Add a passive liveness check that always succeeds when the gateway is up, but fails immediately on a TCP RST/timeout when the process is gone — and stretch the stop deadline so a healthy drain can complete.
Diff scope
Single file:
hermes_squid/stack-compose.yml, +9 lines.Notes / caveats
8087is the gateway dashboard bind (HERMES_DASHBOARD_HOST=0.0.0.0); a successful TCP connect means the gateway event loop is alive. If a future image exposes a dedicated/healthHTTP endpoint, swap the test for["CMD", "wget", "-qO-", "http://127.0.0.1:PORT/health"].stop_grace_periodlives inservices.<name>.stop_grace_periodforcompose-specv3.9; brencher'sdocker stack deployhonours it. Source: Docker docs.Test plan (on prod)
docker service update --with-registry-image): redeployhermes_infraonly.docker service inspect hermes_infra_hermes-gateway | jq '.Spec.TaskTemplate.ContainerSpec.Healthcheck'→ present.docker service ps hermes_infra_hermes-gateway→ task state should report(healthy)./opt/data/logs/gateway.log(hermes_infra_hermes-gateway) for 24 h: the 30sintervalhealthchecks should leave no traces (passive check). Adrain timed out after 0.0sline should not appear.Rollback
Refs:
admin/ansible#26.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.