Production brencher UI reports constant WebSocket reconnects (see
user observation 2026-09-05, fact #5). Prod logs from brencher_brencher-backend (image brencher:auto-74bf5aef-…) show WebSocket /ws [accepted] followed within ~3–10 s by KeyError: <starlette.websockets.WebSocket object …> raised at backend/web.py:175, then connection closed. Same UI reconnecting
after the 5 s backoff. The companion brencher2 instance (one
stack, no SECONDARY_BRENCHER) shows 0 errors over the same window,
confirming that the same code on a single-stack single-source path
is race-free.
Root cause
ws_connections: Dict[WebSocket, Dict[str, Any]] is mutated from
two coroutines that race on the asyncio event loop:
broadcast() — invoked via asyncio.run_coroutine_threadsafe
from the processing thread on every onupdate(). With 7 profiles
on the new brencher this fires ≥10 times per ~100 s cycle, and
the SecondaryManager’s on_update=lambda: self.broadcast_all()
layers additional dispatches on top.
websocket_endpoint() — the FastAPI handler running on the
uvicorn loop. Between await websocket.accept() and self.ws_connections[websocket] = {}, the handler yields. A
concurrent broadcast that was already iterating self.ws_connections.keys() may include the freshly accepted
socket in disconnected (because its own send_text raised
earlier and was logged), then pop removes it before websocket_endpoint writes back. The handler then hits self.ws_connections[websocket]["branches"] = … and raises KeyError. Starlette’s default error path closes the WS.
Fix (this PR — variant setdefault)
Treat ws_connections as a sparse dict: rely on dict.setdefault for atomic insert-or-get, and hold a local state binding throughout the handler so every subsequent write
operates on the same dict object even if a concurrent broadcast
removes and re-adds the entry.
Diff is 12 lines, no API or protocol change:
broadcast(): iterate ws_connections.items(); write to the
local state[event] = message instead of self.ws_connections[websocket][event] = message.
websocket_endpoint(): replace ws_connections[websocket] = {}
with state = ws_connections.setdefault(websocket, {}); replace
the two follow-up writes with state["branches"] = … / state["environments"] = ….
Properties:
A broadcast scheduled before setdefault runs is a no-op for
the freshly accepted client (its first frame is the handler’s
initial send_text, so losing one in-flight broadcast is
benign; the next periodic broadcast at ~10 s re-sends).
Stale state returned by setdefault is overwritten in place
— no lock needed because CPython dict operations are atomic
under the GIL, and we only ever touch the dict’s value binding,
never its key existence.
Code reads cleaner: there’s one state = ws_connections[…]
binding per WS, no self.ws_connections[websocket]["x"]
chains.
Companion PRs (variants 3 and 4 — same race, different shape)
fix/ws-asyncio-lock — explicit asyncio.Lock around the
dict; serializes mutation but not send_text.
All three solve the same root cause with different trade-offs. The
recommended merge order is this one (smallest diff, lowest review
surface). Promote to Lock or Queue only if benchmarking shows
setdefault is insufficient under heavier connection concurrency.
Validation
python3 -m py_compile backend/web.py → OK.
Diff: 12 lines in one file.
Manual verification on production is the user’s call. The prod
logs already show the pre-fix error pattern (brencher_brencher-backend
1.80 WS errors per WS_OPEN), so post-deploy the same log extraction
script (in prod-ws-error-count skill) should drop to 0.
## Symptom
Production brencher UI reports constant WebSocket reconnects (see
user observation 2026-09-05, fact #5). Prod logs from
`brencher_brencher-backend` (image `brencher:auto-74bf5aef-…`) show
`WebSocket /ws [accepted]` followed within ~3–10 s by
`KeyError: <starlette.websockets.WebSocket object …>` raised at
`backend/web.py:175`, then `connection closed`. Same UI reconnecting
after the 5 s backoff. The companion `brencher2` instance (one
stack, no `SECONDARY_BRENCHER`) shows 0 errors over the same window,
confirming that the same code on a single-stack single-source path
is race-free.
## Root cause
`ws_connections: Dict[WebSocket, Dict[str, Any]]` is mutated from
two coroutines that race on the asyncio event loop:
- `broadcast()` — invoked via `asyncio.run_coroutine_threadsafe`
from the processing thread on every `onupdate()`. With 7 profiles
on the new brencher this fires ≥10 times per ~100 s cycle, and
the SecondaryManager’s `on_update=lambda: self.broadcast_all()`
layers additional dispatches on top.
- `websocket_endpoint()` — the FastAPI handler running on the
uvicorn loop. Between `await websocket.accept()` and
`self.ws_connections[websocket] = {}`, the handler yields. A
concurrent broadcast that was already iterating
`self.ws_connections.keys()` may include the freshly accepted
socket in `disconnected` (because its own `send_text` raised
earlier and was logged), then `pop` removes it before
`websocket_endpoint` writes back. The handler then hits
`self.ws_connections[websocket]["branches"] = …` and raises
`KeyError`. Starlette’s default error path closes the WS.
## Fix (this PR — variant `setdefault`)
Treat `ws_connections` as a sparse dict: rely on
`dict.setdefault` for atomic insert-or-get, and hold a local
`state` binding throughout the handler so every subsequent write
operates on the same dict object even if a concurrent broadcast
removes and re-adds the entry.
Diff is 12 lines, no API or protocol change:
- `broadcast()`: iterate `ws_connections.items()`; write to the
local `state[event] = message` instead of
`self.ws_connections[websocket][event] = message`.
- `websocket_endpoint()`: replace `ws_connections[websocket] = {}`
with `state = ws_connections.setdefault(websocket, {})`; replace
the two follow-up writes with `state["branches"] = …` /
`state["environments"] = …`.
Properties:
- A broadcast scheduled before `setdefault` runs is a no-op for
the freshly accepted client (its first frame is the handler’s
initial `send_text`, so losing one in-flight broadcast is
benign; the next periodic broadcast at `~10 s` re-sends).
- Stale `state` returned by `setdefault` is overwritten in place
— no lock needed because CPython dict operations are atomic
under the GIL, and we only ever touch the dict’s value binding,
never its key existence.
- Code reads cleaner: there’s one `state = ws_connections[…]`
binding per WS, no `self.ws_connections[websocket]["x"]`
chains.
## Companion PRs (variants 3 and 4 — same race, different shape)
- `fix/ws-asyncio-lock` — explicit `asyncio.Lock` around the
dict; serializes mutation but not `send_text`.
- `fix/ws-writer-loop` — single-writer task consuming an
`asyncio.Queue`; broadcast becomes inherently sequential.
All three solve the same root cause with different trade-offs. The
recommended merge order is this one (smallest diff, lowest review
surface). Promote to `Lock` or `Queue` only if benchmarking shows
setdefault is insufficient under heavier connection concurrency.
## Validation
- `python3 -m py_compile backend/web.py` → OK.
- Diff: 12 lines in one file.
Manual verification on production is the user’s call. The prod
logs already show the pre-fix error pattern (`brencher_brencher-backend`
1.80 WS errors per WS_OPEN), so post-deploy the same log extraction
script (in `prod-ws-error-count` skill) should drop to 0.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Remove checkPing1 (UrlCheck against the gateway /api/status) from the
hermes environment pipeline and comment the corresponding object out, so
deploys to the hermes stack no longer fail the pipeline on a flaky health
probe.
Mirrors the change already applied to backend/configs/hermes_infra.py
(checkPing1 disabled there too). UrlCheck is no longer referenced from
this config, so drop it from the import.
`ws_connections` is mutated concurrently by `broadcast()` (called from
`run_coroutine_threadsafe` in the processing thread) and by
`websocket_endpoint` (the FastAPI handler running on the uvicorn loop).
When a UI client connects, `await websocket.accept()` yields control,
and a concurrent broadcast may execute `ws_connections.pop(ws, None)`
between `accept()` and the websocket_endpoint handler storing its
initial state, raising KeyError inside the handler. Starlette then
closes the WS, and the UI reconnects after the 5s backoff.
Iterate ws_connections.items() and write to the local `state` binding
in broadcast(), and use `setdefault(websocket, {})` in
`websocket_endpoint` so that the handler never writes to a key that
the cleanup pass of a concurrent broadcast has already removed.
This eliminates the WS Error / KeyError traces observed in prod logs
without changing protocol semantics: a message aimed at a client that
just disconnected is silently skipped, and the next periodic broadcast
restores the client state.
Self-standing fix, no API changes. Companion PRs:
- fix/ws-asyncio-lock (variant 3)
- fix/ws-writer-loop (variant 4)
The three fixes solve the same race with different tradeoffs; keep
them as parallel candidates and pick one after production validation.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Symptom
Production brencher UI reports constant WebSocket reconnects (see
user observation 2026-09-05, fact #5). Prod logs from
brencher_brencher-backend(imagebrencher:auto-74bf5aef-…) showWebSocket /ws [accepted]followed within ~3–10 s byKeyError: <starlette.websockets.WebSocket object …>raised atbackend/web.py:175, thenconnection closed. Same UI reconnectingafter the 5 s backoff. The companion
brencher2instance (onestack, no
SECONDARY_BRENCHER) shows 0 errors over the same window,confirming that the same code on a single-stack single-source path
is race-free.
Root cause
ws_connections: Dict[WebSocket, Dict[str, Any]]is mutated fromtwo coroutines that race on the asyncio event loop:
broadcast()— invoked viaasyncio.run_coroutine_threadsafefrom the processing thread on every
onupdate(). With 7 profileson the new brencher this fires ≥10 times per ~100 s cycle, and
the SecondaryManager’s
on_update=lambda: self.broadcast_all()layers additional dispatches on top.
websocket_endpoint()— the FastAPI handler running on theuvicorn loop. Between
await websocket.accept()andself.ws_connections[websocket] = {}, the handler yields. Aconcurrent broadcast that was already iterating
self.ws_connections.keys()may include the freshly acceptedsocket in
disconnected(because its ownsend_textraisedearlier and was logged), then
popremoves it beforewebsocket_endpointwrites back. The handler then hitsself.ws_connections[websocket]["branches"] = …and raisesKeyError. Starlette’s default error path closes the WS.Fix (this PR — variant
setdefault)Treat
ws_connectionsas a sparse dict: rely ondict.setdefaultfor atomic insert-or-get, and hold a localstatebinding throughout the handler so every subsequent writeoperates on the same dict object even if a concurrent broadcast
removes and re-adds the entry.
Diff is 12 lines, no API or protocol change:
broadcast(): iteratews_connections.items(); write to thelocal
state[event] = messageinstead ofself.ws_connections[websocket][event] = message.websocket_endpoint(): replacews_connections[websocket] = {}with
state = ws_connections.setdefault(websocket, {}); replacethe two follow-up writes with
state["branches"] = …/state["environments"] = ….Properties:
setdefaultruns is a no-op forthe freshly accepted client (its first frame is the handler’s
initial
send_text, so losing one in-flight broadcast isbenign; the next periodic broadcast at
~10 sre-sends).statereturned bysetdefaultis overwritten in place— no lock needed because CPython dict operations are atomic
under the GIL, and we only ever touch the dict’s value binding,
never its key existence.
state = ws_connections[…]binding per WS, no
self.ws_connections[websocket]["x"]chains.
Companion PRs (variants 3 and 4 — same race, different shape)
fix/ws-asyncio-lock— explicitasyncio.Lockaround thedict; serializes mutation but not
send_text.fix/ws-writer-loop— single-writer task consuming anasyncio.Queue; broadcast becomes inherently sequential.All three solve the same root cause with different trade-offs. The
recommended merge order is this one (smallest diff, lowest review
surface). Promote to
LockorQueueonly if benchmarking showssetdefault is insufficient under heavier connection concurrency.
Validation
python3 -m py_compile backend/web.py→ OK.Manual verification on production is the user’s call. The prod
logs already show the pre-fix error pattern (
brencher_brencher-backend1.80 WS errors per WS_OPEN), so post-deploy the same log extraction
script (in
prod-ws-error-countskill) should drop to 0.🤖 Generated with Claude Code
`ws_connections` is mutated concurrently by `broadcast()` (called from `run_coroutine_threadsafe` in the processing thread) and by `websocket_endpoint` (the FastAPI handler running on the uvicorn loop). When a UI client connects, `await websocket.accept()` yields control, and a concurrent broadcast may execute `ws_connections.pop(ws, None)` between `accept()` and the websocket_endpoint handler storing its initial state, raising KeyError inside the handler. Starlette then closes the WS, and the UI reconnects after the 5s backoff. Iterate ws_connections.items() and write to the local `state` binding in broadcast(), and use `setdefault(websocket, {})` in `websocket_endpoint` so that the handler never writes to a key that the cleanup pass of a concurrent broadcast has already removed. This eliminates the WS Error / KeyError traces observed in prod logs without changing protocol semantics: a message aimed at a client that just disconnected is silently skipped, and the next periodic broadcast restores the client state. Self-standing fix, no API changes. Companion PRs: - fix/ws-asyncio-lock (variant 3) - fix/ws-writer-loop (variant 4) The three fixes solve the same race with different tradeoffs; keep them as parallel candidates and pick one after production validation.