fix(configs/hermes_infra): wire unmerge into state holder for restart recovery #61

Merged
admin merged 1 commits from hermes/brencher:fix/hermes_infra-unmerge-recovery into main 2026-09-12 00:15:40 +02:00
Contributor

Symptom

After every brencher container restart, the hermes_infra environment
spams the logs with:

processing - ERROR - Error processing release hermes_infra, job CheckoutMerged: Empty branches set
processing - ERROR - Error processing release hermes_infra, job DockerSwarmDeploy: Empty branches set

hermes_infra is the only environment affected. All other six configs
(brencher, brencher2, hermes, immich, gmail_mcp,
torrserv_proxy) recover cleanly. Frequency observed in the prod logs
of brencher-new (auto-52692a75-…) after the 2026-09-11 redeploy:
~20 Empty branches set errors per ~15-minute window, all attributed
to hermes_infra.

Root cause

SharedStateHolderInMemory.progress() (backend/steps/shared_state.py)
falls back to self.unmerge.progress() when the in-memory state is
empty. The fallback exists to recover after a brencher container
restart: the deployed version (e.g. auto-6848d114) is read from
compose service labels via DockerSwarmCheck, then GitUnmerge
decodes the version into a (branch_name, commit) tuple by walking
the local git refs.

The fallback only runs when the state was constructed with a non-None
unmerge. backend/configs/hermes_infra.py historically passed
unmerge=None (the file has had this since its creation in 57393c9
«hermes (#83)» of 5 Aug 2026). The recovery path therefore becomes a
no-op, state.branches stays empty, and CheckoutMerged.progress()
raises BaseException("Empty branches set") on every processing cycle.

Why only this one config? All six other configs pass
unmerge=unmerge. hermes_infra is the lone outlier; it is also the
only config whose state is built with the explicitly-empty
constructor argument. No commit message or PR comment explains the
choice — the file has had it from day one.

Fix

Pass the existing unmerge step (already constructed two lines above
and registered in the pipeline) into SharedStateHolderInMemory:

-state = SharedStateHolderInMemory(unmerge=None)
+state = SharedStateHolderInMemory(unmerge=unmerge)

A comment block is added explaining why this is wired (recovery
semantics in SharedStateHolderInMemory.progress()).

This brings hermes_infra into line with every other config that
uses SharedStateHolderInMemory. No other file changes.

Why this is correct

  • unmerge = GitUnmerge(clone, dockerSwarmCheck) already exists and
    is a member of the pipeline. Sharing the instance is intended —
    GitUnmerge.progress() reads the deployed compose version via
    dockerSwarmCheck and decodes the auto-<hex> label into a
    (branch_name, commit) tuple by walking the local git refs.
  • The deployed compose version for hermes_infra is auto-6848d114
    (current prod). The brencher bot pushes this via a
    recover/auto-* branch in admin/ansible; the source branch
    feat/docker-socket-proxy-readonly on hermes/ansible points at
    6848d114 and is reachable through the cloned gitea_hermes
    remote.
  • After this change, restart recovery calls unmerge.progress()
    inside state.progress(), populating state.branches with the
    resolved (branch_name, commit) pair so CheckoutMerged and the
    downstream DockerSwarmDeploy step proceed normally.

Risk

Low. The change is one line plus a comment. The other six configs use
this exact pattern and have been stable on prod for months. The only
behavioural change is the unmerge step running twice per cycle
instead of once: once from the pipeline, once from
state.progress() on the empty-state recovery path. Both invocations
operate on the same GitClone working directory and produce identical
results (deterministic decode of the version label); running the
decode twice is already the case for every other config.

Edge case: if unmerge.progress() fails (e.g. the deployed version's
commit is no longer reachable in any cloned remote), state.progress()
now logs Unmerge failed: … from shared_state.py and the state
stays empty — CheckoutMerged then continues to raise
Empty branches set for that env. This is the same fail mode the
current code has for hermes and friends; the new code makes the
failure mode visible (one extra log line) rather than silently
masking it. If the deployed version ever becomes unreachable, a
follow-up can address the underlying reachability rather than this
fallback.

Verification

  • python3 -m py_compile backend/configs/hermes_infra.py → OK
  • Diff: 1 line of code, +14 / -1 in backend/configs/hermes_infra.py
  • Manual: cross-checked against the other six configs in
    backend/configs/, all of which already use unmerge=unmerge
    (brencher.py, brencher2.py, hermes.py, immich.py,
    gmail_mcp.py, torrserv_proxy.py).

Production validation (post-merge, manual): the same brencher-new
container should drop its Empty branches set log volume for the
hermes_infra env from ~20 errors / 15 min to 0, unless the
deployed version becomes unreachable. Pre-merge baseline recorded in
the commit message of this PR.

🤖 Generated with Claude Code

## Symptom After every brencher container restart, the `hermes_infra` environment spams the logs with: ``` processing - ERROR - Error processing release hermes_infra, job CheckoutMerged: Empty branches set processing - ERROR - Error processing release hermes_infra, job DockerSwarmDeploy: Empty branches set ``` `hermes_infra` is the only environment affected. All other six configs (`brencher`, `brencher2`, `hermes`, `immich`, `gmail_mcp`, `torrserv_proxy`) recover cleanly. Frequency observed in the prod logs of `brencher-new` (`auto-52692a75-…`) after the 2026-09-11 redeploy: ~20 `Empty branches set` errors per ~15-minute window, all attributed to `hermes_infra`. ## Root cause `SharedStateHolderInMemory.progress()` (backend/steps/shared_state.py) falls back to `self.unmerge.progress()` when the in-memory state is empty. The fallback exists to recover after a brencher container restart: the deployed `version` (e.g. `auto-6848d114`) is read from compose service labels via `DockerSwarmCheck`, then `GitUnmerge` decodes the version into a `(branch_name, commit)` tuple by walking the local git refs. The fallback only runs when the state was constructed with a non-None `unmerge`. `backend/configs/hermes_infra.py` historically passed `unmerge=None` (the file has had this since its creation in `57393c9` «hermes (#83)» of 5 Aug 2026). The recovery path therefore becomes a no-op, `state.branches` stays empty, and `CheckoutMerged.progress()` raises `BaseException("Empty branches set")` on every processing cycle. Why only this one config? All six other configs pass `unmerge=unmerge`. `hermes_infra` is the lone outlier; it is also the only config whose `state` is built with the explicitly-empty constructor argument. No commit message or PR comment explains the choice — the file has had it from day one. ## Fix Pass the existing `unmerge` step (already constructed two lines above and registered in the pipeline) into `SharedStateHolderInMemory`: ```diff -state = SharedStateHolderInMemory(unmerge=None) +state = SharedStateHolderInMemory(unmerge=unmerge) ``` A comment block is added explaining why this is wired (recovery semantics in `SharedStateHolderInMemory.progress()`). This brings `hermes_infra` into line with every other config that uses `SharedStateHolderInMemory`. No other file changes. ## Why this is correct - `unmerge = GitUnmerge(clone, dockerSwarmCheck)` already exists and is a member of the pipeline. Sharing the instance is intended — `GitUnmerge.progress()` reads the deployed compose version via `dockerSwarmCheck` and decodes the `auto-<hex>` label into a `(branch_name, commit)` tuple by walking the local git refs. - The deployed compose version for `hermes_infra` is `auto-6848d114` (current prod). The brencher bot pushes this via a `recover/auto-*` branch in `admin/ansible`; the source branch `feat/docker-socket-proxy-readonly` on `hermes/ansible` points at `6848d114` and is reachable through the cloned `gitea_hermes` remote. - After this change, restart recovery calls `unmerge.progress()` inside `state.progress()`, populating `state.branches` with the resolved `(branch_name, commit)` pair so `CheckoutMerged` and the downstream `DockerSwarmDeploy` step proceed normally. ## Risk Low. The change is one line plus a comment. The other six configs use this exact pattern and have been stable on prod for months. The only behavioural change is the `unmerge` step running twice per cycle instead of once: once from the pipeline, once from `state.progress()` on the empty-state recovery path. Both invocations operate on the same `GitClone` working directory and produce identical results (deterministic decode of the version label); running the decode twice is already the case for every other config. Edge case: if `unmerge.progress()` fails (e.g. the deployed version's commit is no longer reachable in any cloned remote), `state.progress()` now logs `Unmerge failed: …` from `shared_state.py` and the state stays empty — `CheckoutMerged` then continues to raise `Empty branches set` for that env. This is the same fail mode the current code has for `hermes` and friends; the new code makes the failure mode visible (one extra log line) rather than silently masking it. If the deployed version ever becomes unreachable, a follow-up can address the underlying reachability rather than this fallback. ## Verification - `python3 -m py_compile backend/configs/hermes_infra.py` → OK - Diff: 1 line of code, +14 / -1 in `backend/configs/hermes_infra.py` - Manual: cross-checked against the other six configs in `backend/configs/`, all of which already use `unmerge=unmerge` (`brencher.py`, `brencher2.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`). Production validation (post-merge, manual): the same `brencher-new` container should drop its `Empty branches set` log volume for the `hermes_infra` env from ~20 errors / 15 min to 0, unless the deployed version becomes unreachable. Pre-merge baseline recorded in the commit message of this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
hermes added 1 commit 2026-09-11 18:39:30 +02:00
fix(configs/hermes_infra): wire unmerge into state holder for restart recovery
Python package / build (3.12.3) (pull_request) Successful in 2m24s
Python package / build (3.13) (pull_request) Successful in 2m23s
Python package / build (3.14) (pull_request) Successful in 2m29s
04069fe561
hermes force-pushed fix/hermes_infra-unmerge-recovery from e3bd669fc1 to 04069fe561 2026-09-11 18:39:30 +02:00 Compare
Author
Contributor

Ready to test.

Tracking ticket: #62 — checklist for staging → prod deploy and the verification steps described there. CI is green on the current head (3.12.3 / 3.13 / 3.14).

Ready to test. Tracking ticket: #62 — checklist for staging → prod deploy and the verification steps described there. CI is green on the current head (3.12.3 / 3.13 / 3.14).
admin merged commit 7938f4f0f0 into main 2026-09-12 00:15:40 +02:00
admin deleted branch fix/hermes_infra-unmerge-recovery 2026-09-12 00:15:41 +02:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/brencher#61