Raise the hermes-gateway container's memory limit from 1G to 2G and the Pids limit from 512 to 1024. Reservation follows the limit.
Why
admin/ansible#26 shows:
27 occurrences of WARNING gateway.run: Persisted transcript lagged live cached history … (possible FTS write corruption) on the active Home session (agent:main:telegram:group:-1004417969359:16) in 24h, each with disk=N, memory=N+1.
Two full gateway restarts driven by Received SIGTERM … parent_pid=148 parent_name=s6-supervise (18:45 and 22:53), recovered by Scheduled auto-resume for 1 restart-interrupted session(s).
The 1G limit is the trip wire. A long Home session with kt/sessions/<id> accumulating inline tool results and logs/agent.log writes pushes the gateway past the limit; s6 then SIGTERMs with no healthcheck to fall back on. PR #27 adds the healthcheck and stop_grace_period; this PR raises the bar so the same workload no longer crosses it.
2 GiB gives ~2× current peak (peak observed ~1.5 GiB today via /stats rough read). It is intentionally not 4G so unexpected leaks still get caught.
1024 Pids covers the documented need of ~30 long-lived Pids + a burst of subprocess.Popen calls from tool execution; the previous 512 limit was a real collision risk during cron + tailwind Kanban flush.
Rollback: git revert -m 1 HEAD or docker service update --limit-memory 1G hermes_infra_hermes-gateway.
Test plan
After merging + brencher rolling update:
docker service inspect hermes_infra_hermes-gateway --format '{{.Spec.TaskTemplate.Resources.Limits.MemoryBytes}}' → 2147483648 (2 GiB).
docker stats shows MEM USAGE climbing but not crossing 1.5 GiB during normal load.
tail -F /opt/data/logs/gateway.log for 24 h: no new Persisted transcript lagged lines OR a clear downward trend.
Follow-ups
PR #27 (healthcheck + stop_grace_period) — should be merged and deployed first; this PR only raises the limit.
PR #29 (README pinning the chat ↔ stack mapping) and PR #30 (recovery playbook) are documentation-only and independent.
Refs: admin/ansible#26.
## Summary
Raise the `hermes-gateway` container's memory limit from 1G to 2G and the Pids limit from 512 to 1024. Reservation follows the limit.
## Why
`admin/ansible#26` shows:
- 27 occurrences of `WARNING gateway.run: Persisted transcript lagged live cached history … (possible FTS write corruption)` on the active Home session (`agent:main:telegram:group:-1004417969359:16`) in 24h, each with `disk=N, memory=N+1`.
- Two full gateway restarts driven by `Received SIGTERM … parent_pid=148 parent_name=s6-supervise` (18:45 and 22:53), recovered by `Scheduled auto-resume for 1 restart-interrupted session(s)`.
The 1G limit is the trip wire. A long Home session with `kt/sessions/<id>` accumulating inline tool results and `logs/agent.log` writes pushes the gateway past the limit; `s6` then SIGTERMs with no healthcheck to fall back on. PR #27 adds the healthcheck and stop_grace_period; this PR raises the bar so the same workload no longer crosses it.
## Diff
```yaml
resources:
limits:
- memory: 1G
- pids: 512
+ memory: 2G
+ pids: 1024
reservations:
- memory: 1G
+ memory: 2G
```
## Caveats
- 2 GiB gives ~2× current peak (peak observed ~1.5 GiB today via `/stats` rough read). It is intentionally not 4G so unexpected leaks still get caught.
- 1024 Pids covers the documented need of ~30 long-lived Pids + a burst of `subprocess.Popen` calls from tool execution; the previous 512 limit was a real collision risk during cron + tailwind Kanban flush.
- Rollback: `git revert -m 1 HEAD` or `docker service update --limit-memory 1G hermes_infra_hermes-gateway`.
## Test plan
After merging + brencher rolling update:
1. `docker service inspect hermes_infra_hermes-gateway --format '{{.Spec.TaskTemplate.Resources.Limits.MemoryBytes}}'` → `2147483648` (2 GiB).
2. `docker stats` shows `MEM USAGE` climbing but not crossing 1.5 GiB during normal load.
3. `tail -F /opt/data/logs/gateway.log` for 24 h: no new `Persisted transcript lagged` lines OR a clear downward trend.
## Follow-ups
- PR #27 (healthcheck + stop_grace_period) — should be merged and deployed first; this PR only raises the limit.
- PR #29 (README pinning the chat ↔ stack mapping) and PR #30 (recovery playbook) are documentation-only and independent.
Refs: `admin/ansible#26`.
Long-running Home sessions push the gateway past 1G RSS during FTS
writeback (27 'Persisted transcript lagged' WARNs observed today alone).
2G keeps room for context growth and concurrent kanban/heartbeat loops
without dropping back to 'Scheduled auto-resume' on every drain.
Refs admin/ansible#26, follows PR #27 (healthcheck+stop_grace_period).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Raise the
hermes-gatewaycontainer's memory limit from 1G to 2G and the Pids limit from 512 to 1024. Reservation follows the limit.Why
admin/ansible#26shows:WARNING gateway.run: Persisted transcript lagged live cached history … (possible FTS write corruption)on the active Home session (agent:main:telegram:group:-1004417969359:16) in 24h, each withdisk=N, memory=N+1.Received SIGTERM … parent_pid=148 parent_name=s6-supervise(18:45 and 22:53), recovered byScheduled auto-resume for 1 restart-interrupted session(s).The 1G limit is the trip wire. A long Home session with
kt/sessions/<id>accumulating inline tool results andlogs/agent.logwrites pushes the gateway past the limit;s6then SIGTERMs with no healthcheck to fall back on. PR #27 adds the healthcheck and stop_grace_period; this PR raises the bar so the same workload no longer crosses it.Diff
Caveats
/statsrough read). It is intentionally not 4G so unexpected leaks still get caught.subprocess.Popencalls from tool execution; the previous 512 limit was a real collision risk during cron + tailwind Kanban flush.git revert -m 1 HEADordocker service update --limit-memory 1G hermes_infra_hermes-gateway.Test plan
After merging + brencher rolling update:
docker service inspect hermes_infra_hermes-gateway --format '{{.Spec.TaskTemplate.Resources.Limits.MemoryBytes}}'→2147483648(2 GiB).docker statsshowsMEM USAGEclimbing but not crossing 1.5 GiB during normal load.tail -F /opt/data/logs/gateway.logfor 24 h: no newPersisted transcript laggedlines OR a clear downward trend.Follow-ups
Refs:
admin/ansible#26.View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.