hermes_infra is the only environment affected. All other six configs
(brencher, brencher2, hermes, immich, gmail_mcp, torrserv_proxy) recover cleanly. Frequency observed in the prod logs
of brencher-new (auto-52692a75-…) after the 2026-09-11 redeploy:
~20 Empty branches set errors per ~15-minute window, all attributed
to hermes_infra.
Root cause
SharedStateHolderInMemory.progress() (backend/steps/shared_state.py)
falls back to self.unmerge.progress() when the in-memory state is
empty. The fallback exists to recover after a brencher container
restart: the deployed version (e.g. auto-6848d114) is read from
compose service labels via DockerSwarmCheck, then GitUnmerge
decodes the version into a (branch_name, commit) tuple by walking
the local git refs.
The fallback only runs when the state was constructed with a non-None unmerge. backend/configs/hermes_infra.py historically passed unmerge=None (the file has had this since its creation in 57393c9
«hermes (#83)» of 5 Aug 2026). The recovery path therefore becomes a
no-op, state.branches stays empty, and CheckoutMerged.progress()
raises BaseException("Empty branches set") on every processing cycle.
Why only this one config? All six other configs pass unmerge=unmerge. hermes_infra is the lone outlier; it is also the
only config whose state is built with the explicitly-empty
constructor argument. No commit message or PR comment explains the
choice — the file has had it from day one.
Fix
Pass the existing unmerge step (already constructed two lines above
and registered in the pipeline) into SharedStateHolderInMemory:
A comment block is added explaining why this is wired (recovery
semantics in SharedStateHolderInMemory.progress()).
This brings hermes_infra into line with every other config that
uses SharedStateHolderInMemory. No other file changes.
Why this is correct
unmerge = GitUnmerge(clone, dockerSwarmCheck) already exists and
is a member of the pipeline. Sharing the instance is intended — GitUnmerge.progress() reads the deployed compose version via dockerSwarmCheck and decodes the auto-<hex> label into a (branch_name, commit) tuple by walking the local git refs.
The deployed compose version for hermes_infra is auto-6848d114
(current prod). The brencher bot pushes this via a recover/auto-* branch in admin/ansible; the source branch feat/docker-socket-proxy-readonly on hermes/ansible points at 6848d114 and is reachable through the cloned gitea_hermes
remote.
After this change, restart recovery calls unmerge.progress()
inside state.progress(), populating state.branches with the
resolved (branch_name, commit) pair so CheckoutMerged and the
downstream DockerSwarmDeploy step proceed normally.
Risk
Low. The change is one line plus a comment. The other six configs use
this exact pattern and have been stable on prod for months. The only
behavioural change is the unmerge step running twice per cycle
instead of once: once from the pipeline, once from state.progress() on the empty-state recovery path. Both invocations
operate on the same GitClone working directory and produce identical
results (deterministic decode of the version label); running the
decode twice is already the case for every other config.
Edge case: if unmerge.progress() fails (e.g. the deployed version's
commit is no longer reachable in any cloned remote), state.progress()
now logs Unmerge failed: … from shared_state.py and the state
stays empty — CheckoutMerged then continues to raise Empty branches set for that env. This is the same fail mode the
current code has for hermes and friends; the new code makes the
failure mode visible (one extra log line) rather than silently
masking it. If the deployed version ever becomes unreachable, a
follow-up can address the underlying reachability rather than this
fallback.
Verification
python3 -m py_compile backend/configs/hermes_infra.py → OK
Diff: 1 line of code, +14 / -1 in backend/configs/hermes_infra.py
Manual: cross-checked against the other six configs in backend/configs/, all of which already use unmerge=unmerge
(brencher.py, brencher2.py, hermes.py, immich.py, gmail_mcp.py, torrserv_proxy.py).
Production validation (post-merge, manual): the same brencher-new
container should drop its Empty branches set log volume for the hermes_infra env from ~20 errors / 15 min to 0, unless the
deployed version becomes unreachable. Pre-merge baseline recorded in
the commit message of this PR.
## Symptom
After every brencher container restart, the `hermes_infra` environment
spams the logs with:
```
processing - ERROR - Error processing release hermes_infra, job CheckoutMerged: Empty branches set
processing - ERROR - Error processing release hermes_infra, job DockerSwarmDeploy: Empty branches set
```
`hermes_infra` is the only environment affected. All other six configs
(`brencher`, `brencher2`, `hermes`, `immich`, `gmail_mcp`,
`torrserv_proxy`) recover cleanly. Frequency observed in the prod logs
of `brencher-new` (`auto-52692a75-…`) after the 2026-09-11 redeploy:
~20 `Empty branches set` errors per ~15-minute window, all attributed
to `hermes_infra`.
## Root cause
`SharedStateHolderInMemory.progress()` (backend/steps/shared_state.py)
falls back to `self.unmerge.progress()` when the in-memory state is
empty. The fallback exists to recover after a brencher container
restart: the deployed `version` (e.g. `auto-6848d114`) is read from
compose service labels via `DockerSwarmCheck`, then `GitUnmerge`
decodes the version into a `(branch_name, commit)` tuple by walking
the local git refs.
The fallback only runs when the state was constructed with a non-None
`unmerge`. `backend/configs/hermes_infra.py` historically passed
`unmerge=None` (the file has had this since its creation in `57393c9`
«hermes (#83)» of 5 Aug 2026). The recovery path therefore becomes a
no-op, `state.branches` stays empty, and `CheckoutMerged.progress()`
raises `BaseException("Empty branches set")` on every processing cycle.
Why only this one config? All six other configs pass
`unmerge=unmerge`. `hermes_infra` is the lone outlier; it is also the
only config whose `state` is built with the explicitly-empty
constructor argument. No commit message or PR comment explains the
choice — the file has had it from day one.
## Fix
Pass the existing `unmerge` step (already constructed two lines above
and registered in the pipeline) into `SharedStateHolderInMemory`:
```diff
-state = SharedStateHolderInMemory(unmerge=None)
+state = SharedStateHolderInMemory(unmerge=unmerge)
```
A comment block is added explaining why this is wired (recovery
semantics in `SharedStateHolderInMemory.progress()`).
This brings `hermes_infra` into line with every other config that
uses `SharedStateHolderInMemory`. No other file changes.
## Why this is correct
- `unmerge = GitUnmerge(clone, dockerSwarmCheck)` already exists and
is a member of the pipeline. Sharing the instance is intended —
`GitUnmerge.progress()` reads the deployed compose version via
`dockerSwarmCheck` and decodes the `auto-<hex>` label into a
`(branch_name, commit)` tuple by walking the local git refs.
- The deployed compose version for `hermes_infra` is `auto-6848d114`
(current prod). The brencher bot pushes this via a
`recover/auto-*` branch in `admin/ansible`; the source branch
`feat/docker-socket-proxy-readonly` on `hermes/ansible` points at
`6848d114` and is reachable through the cloned `gitea_hermes`
remote.
- After this change, restart recovery calls `unmerge.progress()`
inside `state.progress()`, populating `state.branches` with the
resolved `(branch_name, commit)` pair so `CheckoutMerged` and the
downstream `DockerSwarmDeploy` step proceed normally.
## Risk
Low. The change is one line plus a comment. The other six configs use
this exact pattern and have been stable on prod for months. The only
behavioural change is the `unmerge` step running twice per cycle
instead of once: once from the pipeline, once from
`state.progress()` on the empty-state recovery path. Both invocations
operate on the same `GitClone` working directory and produce identical
results (deterministic decode of the version label); running the
decode twice is already the case for every other config.
Edge case: if `unmerge.progress()` fails (e.g. the deployed version's
commit is no longer reachable in any cloned remote), `state.progress()`
now logs `Unmerge failed: …` from `shared_state.py` and the state
stays empty — `CheckoutMerged` then continues to raise
`Empty branches set` for that env. This is the same fail mode the
current code has for `hermes` and friends; the new code makes the
failure mode visible (one extra log line) rather than silently
masking it. If the deployed version ever becomes unreachable, a
follow-up can address the underlying reachability rather than this
fallback.
## Verification
- `python3 -m py_compile backend/configs/hermes_infra.py` → OK
- Diff: 1 line of code, +14 / -1 in `backend/configs/hermes_infra.py`
- Manual: cross-checked against the other six configs in
`backend/configs/`, all of which already use `unmerge=unmerge`
(`brencher.py`, `brencher2.py`, `hermes.py`, `immich.py`,
`gmail_mcp.py`, `torrserv_proxy.py`).
Production validation (post-merge, manual): the same `brencher-new`
container should drop its `Empty branches set` log volume for the
`hermes_infra` env from ~20 errors / 15 min to 0, unless the
deployed version becomes unreachable. Pre-merge baseline recorded in
the commit message of this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Tracking ticket: #62 — checklist for staging → prod deploy and the verification steps described there. CI is green on the current head (3.12.3 / 3.13 / 3.14).
Ready to test.
Tracking ticket: #62 — checklist for staging → prod deploy and the verification steps described there. CI is green on the current head (3.12.3 / 3.13 / 3.14).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Symptom
After every brencher container restart, the
hermes_infraenvironmentspams the logs with:
hermes_infrais the only environment affected. All other six configs(
brencher,brencher2,hermes,immich,gmail_mcp,torrserv_proxy) recover cleanly. Frequency observed in the prod logsof
brencher-new(auto-52692a75-…) after the 2026-09-11 redeploy:~20
Empty branches seterrors per ~15-minute window, all attributedto
hermes_infra.Root cause
SharedStateHolderInMemory.progress()(backend/steps/shared_state.py)falls back to
self.unmerge.progress()when the in-memory state isempty. The fallback exists to recover after a brencher container
restart: the deployed
version(e.g.auto-6848d114) is read fromcompose service labels via
DockerSwarmCheck, thenGitUnmergedecodes the version into a
(branch_name, commit)tuple by walkingthe local git refs.
The fallback only runs when the state was constructed with a non-None
unmerge.backend/configs/hermes_infra.pyhistorically passedunmerge=None(the file has had this since its creation in57393c9«hermes (#83)» of 5 Aug 2026). The recovery path therefore becomes a
no-op,
state.branchesstays empty, andCheckoutMerged.progress()raises
BaseException("Empty branches set")on every processing cycle.Why only this one config? All six other configs pass
unmerge=unmerge.hermes_infrais the lone outlier; it is also theonly config whose
stateis built with the explicitly-emptyconstructor argument. No commit message or PR comment explains the
choice — the file has had it from day one.
Fix
Pass the existing
unmergestep (already constructed two lines aboveand registered in the pipeline) into
SharedStateHolderInMemory:A comment block is added explaining why this is wired (recovery
semantics in
SharedStateHolderInMemory.progress()).This brings
hermes_infrainto line with every other config thatuses
SharedStateHolderInMemory. No other file changes.Why this is correct
unmerge = GitUnmerge(clone, dockerSwarmCheck)already exists andis a member of the pipeline. Sharing the instance is intended —
GitUnmerge.progress()reads the deployed compose version viadockerSwarmCheckand decodes theauto-<hex>label into a(branch_name, commit)tuple by walking the local git refs.hermes_infraisauto-6848d114(current prod). The brencher bot pushes this via a
recover/auto-*branch inadmin/ansible; the source branchfeat/docker-socket-proxy-readonlyonhermes/ansiblepoints at6848d114and is reachable through the clonedgitea_hermesremote.
unmerge.progress()inside
state.progress(), populatingstate.brancheswith theresolved
(branch_name, commit)pair soCheckoutMergedand thedownstream
DockerSwarmDeploystep proceed normally.Risk
Low. The change is one line plus a comment. The other six configs use
this exact pattern and have been stable on prod for months. The only
behavioural change is the
unmergestep running twice per cycleinstead of once: once from the pipeline, once from
state.progress()on the empty-state recovery path. Both invocationsoperate on the same
GitCloneworking directory and produce identicalresults (deterministic decode of the version label); running the
decode twice is already the case for every other config.
Edge case: if
unmerge.progress()fails (e.g. the deployed version'scommit is no longer reachable in any cloned remote),
state.progress()now logs
Unmerge failed: …fromshared_state.pyand the statestays empty —
CheckoutMergedthen continues to raiseEmpty branches setfor that env. This is the same fail mode thecurrent code has for
hermesand friends; the new code makes thefailure mode visible (one extra log line) rather than silently
masking it. If the deployed version ever becomes unreachable, a
follow-up can address the underlying reachability rather than this
fallback.
Verification
python3 -m py_compile backend/configs/hermes_infra.py→ OKbackend/configs/hermes_infra.pybackend/configs/, all of which already useunmerge=unmerge(
brencher.py,brencher2.py,hermes.py,immich.py,gmail_mcp.py,torrserv_proxy.py).Production validation (post-merge, manual): the same
brencher-newcontainer should drop its
Empty branches setlog volume for thehermes_infraenv from ~20 errors / 15 min to 0, unless thedeployed version becomes unreachable. Pre-merge baseline recorded in
the commit message of this PR.
🤖 Generated with Claude Code
e3bd669fc1to04069fe561Ready to test.
Tracking ticket: #62 — checklist for staging → prod deploy and the verification steps described there. CI is green on the current head (3.12.3 / 3.13 / 3.14).