fix(hermes_infra): raise hermes-gateway memory limit to 2G and pids to 1024 #28

Open
hermes wants to merge 1 commits from hermes/ansible:fix/hermes-gateway-memory-pids into master
Contributor

Summary

Raise the hermes-gateway container's memory limit from 1G to 2G and the Pids limit from 512 to 1024. Reservation follows the limit.

Why

admin/ansible#26 shows:

  • 27 occurrences of WARNING gateway.run: Persisted transcript lagged live cached history … (possible FTS write corruption) on the active Home session (agent:main:telegram:group:-1004417969359:16) in 24h, each with disk=N, memory=N+1.
  • Two full gateway restarts driven by Received SIGTERM … parent_pid=148 parent_name=s6-supervise (18:45 and 22:53), recovered by Scheduled auto-resume for 1 restart-interrupted session(s).

The 1G limit is the trip wire. A long Home session with kt/sessions/<id> accumulating inline tool results and logs/agent.log writes pushes the gateway past the limit; s6 then SIGTERMs with no healthcheck to fall back on. PR #27 adds the healthcheck and stop_grace_period; this PR raises the bar so the same workload no longer crosses it.

Diff

resources:
  limits:
-    memory: 1G
-    pids: 512
+    memory: 2G
+    pids: 1024
   reservations:
-    memory: 1G
+    memory: 2G

Caveats

  • 2 GiB gives ~2× current peak (peak observed ~1.5 GiB today via /stats rough read). It is intentionally not 4G so unexpected leaks still get caught.
  • 1024 Pids covers the documented need of ~30 long-lived Pids + a burst of subprocess.Popen calls from tool execution; the previous 512 limit was a real collision risk during cron + tailwind Kanban flush.
  • Rollback: git revert -m 1 HEAD or docker service update --limit-memory 1G hermes_infra_hermes-gateway.

Test plan

After merging + brencher rolling update:

  1. docker service inspect hermes_infra_hermes-gateway --format '{{.Spec.TaskTemplate.Resources.Limits.MemoryBytes}}' → 2147483648 (2 GiB).
  2. docker stats shows MEM USAGE climbing but not crossing 1.5 GiB during normal load.
  3. tail -F /opt/data/logs/gateway.log for 24 h: no new Persisted transcript lagged lines OR a clear downward trend.

Follow-ups

  • PR #27 (healthcheck + stop_grace_period) — should be merged and deployed first; this PR only raises the limit.
  • PR #29 (README pinning the chat ↔ stack mapping) and PR #30 (recovery playbook) are documentation-only and independent.

Refs: admin/ansible#26.

## Summary Raise the `hermes-gateway` container's memory limit from 1G to 2G and the Pids limit from 512 to 1024. Reservation follows the limit. ## Why `admin/ansible#26` shows: - 27 occurrences of `WARNING gateway.run: Persisted transcript lagged live cached history … (possible FTS write corruption)` on the active Home session (`agent:main:telegram:group:-1004417969359:16`) in 24h, each with `disk=N, memory=N+1`. - Two full gateway restarts driven by `Received SIGTERM … parent_pid=148 parent_name=s6-supervise` (18:45 and 22:53), recovered by `Scheduled auto-resume for 1 restart-interrupted session(s)`. The 1G limit is the trip wire. A long Home session with `kt/sessions/<id>` accumulating inline tool results and `logs/agent.log` writes pushes the gateway past the limit; `s6` then SIGTERMs with no healthcheck to fall back on. PR #27 adds the healthcheck and stop_grace_period; this PR raises the bar so the same workload no longer crosses it. ## Diff ```yaml resources: limits: - memory: 1G - pids: 512 + memory: 2G + pids: 1024 reservations: - memory: 1G + memory: 2G ``` ## Caveats - 2 GiB gives ~2× current peak (peak observed ~1.5 GiB today via `/stats` rough read). It is intentionally not 4G so unexpected leaks still get caught. - 1024 Pids covers the documented need of ~30 long-lived Pids + a burst of `subprocess.Popen` calls from tool execution; the previous 512 limit was a real collision risk during cron + tailwind Kanban flush. - Rollback: `git revert -m 1 HEAD` or `docker service update --limit-memory 1G hermes_infra_hermes-gateway`. ## Test plan After merging + brencher rolling update: 1. `docker service inspect hermes_infra_hermes-gateway --format '{{.Spec.TaskTemplate.Resources.Limits.MemoryBytes}}'` → `2147483648` (2 GiB). 2. `docker stats` shows `MEM USAGE` climbing but not crossing 1.5 GiB during normal load. 3. `tail -F /opt/data/logs/gateway.log` for 24 h: no new `Persisted transcript lagged` lines OR a clear downward trend. ## Follow-ups - PR #27 (healthcheck + stop_grace_period) — should be merged and deployed first; this PR only raises the limit. - PR #29 (README pinning the chat ↔ stack mapping) and PR #30 (recovery playbook) are documentation-only and independent. Refs: `admin/ansible#26`.
hermes added 1 commit 2026-08-30 23:41:53 +02:00
Long-running Home sessions push the gateway past 1G RSS during FTS
writeback (27 'Persisted transcript lagged' WARNs observed today alone).
2G keeps room for context growth and concurrent kanban/heartbeat loops
without dropping back to 'Scheduled auto-resume' on every drain.

Refs admin/ansible#26, follows PR #27 (healthcheck+stop_grace_period).
Checking for merge conflicts…
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u https://gitea.rudolf.keenetic.link/hermes/ansible fix/hermes-gateway-memory-pids:hermes-fix/hermes-gateway-memory-pids
git checkout hermes-fix/hermes-gateway-memory-pids
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/ansible#28