Same as #58: production brencher_brencher-backend (image auto-74bf5aef-…)
UI reconnect loop every ~5 s. Logs show WebSocket /ws [accepted] followed
by KeyError: <starlette.websockets.WebSocket object …> at web.py:175,
closing the connection. Companion brencher2 (one stack, no SECONDARY_BRENCHER)
shows 0 errors — same code is race-free without contention.
Root cause
Documented in #58. ws_connections is mutated from two coroutines:
broadcast() — invoked via asyncio.run_coroutine_threadsafe from the
processing thread on every onupdate(). With 7 profiles + the
SecondaryManager callback, it runs ~10 times per ~100 s cycle.
websocket_endpoint() — FastAPI handler on the uvicorn loop. Between await websocket.accept() and writing per-channel state, the handler
yields control; a concurrent broadcast can pop the socket from the dict
before the handler records its initial state.
The setdefault variant (#58) keeps the broadcast write atomic but still
has a window where the handler writes to a state object that has already
been detached from ws_connections. The next broadcast re-registers a
fresh state, with the duplicate-suppression guarantee now scoped per
re-registration rather than per channel.
Fix (this PR — variant asyncio.Lock)
Make every read, write, and removal of ws_connections go through a
single asyncio.Lock. Hold the lock only while touching the dict —
never across an await. This turns three separate ad-hoc mutations into
one declared mutation surface (async with self._ws_lock:).
In broadcast():
async with self._ws_lock: snapshots (websocket, state) tuples,
filters out the ones already up-to-date, and exits the block.
send_text runs outside the lock so concurrent broadcasts still
pipeline their frames in parallel (no global serialization of network IO).
After all sends, a second async with self._ws_lock: pops the dead
sockets in a single critical section.
In websocket_endpoint():
async with self._ws_lock: inserts the new socket immediately after accept().
After the initial-frame send_text burst, the handler takes the lock
again to read-back the (possibly-evicted-during-await) state. If the
state is gone (get returns None), the handler returns gracefully —
the very next periodic broadcast will repopulate it.
In disconnect / exception handlers, pop(websocket, None) is wrapped
in async with self._ws_lock:.
Diff stats: 35 insertions(+), 12 deletions(-) in backend/web.py.
Compile-checked.
Companion PRs
#58 (fix/ws-setdefault-and-broadcast) — variant 2, smallest diff,
atomic via dict.setdefault. Less explicit about who coordinates,
but cheaper to review and reason about.
fix/ws-writer-loop — variant 4, single-writer + asyncio.Queue.
Removes all cross-task dict access by funnelling through one
consumer task. Larger change; useful if the lock ever shows up in
profiling.
Merge order I would recommend: #58 first (smallest blast radius if it
turns out to over-fix in some edge case). This PR keeps the explicit
locking pattern as the alternative for teams that prefer mutually
exclusive mutations over atomic dict primitives.
## Symptom
Same as #58: production `brencher_brencher-backend` (image `auto-74bf5aef-…`)
UI reconnect loop every ~5 s. Logs show `WebSocket /ws [accepted]` followed
by `KeyError: <starlette.websockets.WebSocket object …>` at `web.py:175`,
closing the connection. Companion `brencher2` (one stack, no `SECONDARY_BRENCHER`)
shows 0 errors — same code is race-free without contention.
## Root cause
Documented in #58. `ws_connections` is mutated from two coroutines:
- `broadcast()` — invoked via `asyncio.run_coroutine_threadsafe` from the
processing thread on every `onupdate()`. With 7 profiles + the
SecondaryManager callback, it runs ~10 times per ~100 s cycle.
- `websocket_endpoint()` — FastAPI handler on the uvicorn loop. Between
`await websocket.accept()` and writing per-channel state, the handler
yields control; a concurrent broadcast can pop the socket from the dict
before the handler records its initial state.
The setdefault variant (#58) keeps the broadcast write atomic but still
has a window where the handler writes to a state object that has already
been detached from `ws_connections`. The next broadcast re-registers a
fresh state, with the duplicate-suppression guarantee now scoped per
re-registration rather than per channel.
## Fix (this PR — variant `asyncio.Lock`)
Make every read, write, and removal of `ws_connections` go through a
single `asyncio.Lock`. Hold the lock only while touching the dict —
never across an `await`. This turns three separate ad-hoc mutations into
one declared mutation surface (`async with self._ws_lock:`).
In `broadcast()`:
- `async with self._ws_lock:` snapshots `(websocket, state)` tuples,
filters out the ones already up-to-date, and exits the block.
- `send_text` runs outside the lock so concurrent broadcasts still
pipeline their frames in parallel (no global serialization of network IO).
- After all sends, a second `async with self._ws_lock:` pops the dead
sockets in a single critical section.
In `websocket_endpoint()`:
- `async with self._ws_lock:` inserts the new socket immediately after
`accept()`.
- After the initial-frame `send_text` burst, the handler takes the lock
again to read-back the (possibly-evicted-during-await) state. If the
state is gone (`get` returns None), the handler returns gracefully —
the very next periodic broadcast will repopulate it.
In disconnect / exception handlers, `pop(websocket, None)` is wrapped
in `async with self._ws_lock:`.
Diff stats: 35 insertions(+), 12 deletions(-) in `backend/web.py`.
Compile-checked.
## Companion PRs
- #58 (`fix/ws-setdefault-and-broadcast`) — variant 2, smallest diff,
atomic via `dict.setdefault`. Less explicit about who coordinates,
but cheaper to review and reason about.
- `fix/ws-writer-loop` — variant 4, single-writer + `asyncio.Queue`.
Removes all cross-task dict access by funnelling through one
consumer task. Larger change; useful if the lock ever shows up in
profiling.
Merge order I would recommend: #58 first (smallest blast radius if it
turns out to over-fix in some edge case). This PR keeps the explicit
locking pattern as the alternative for teams that prefer mutually
exclusive mutations over atomic dict primitives.
## Validation
- `python3 -m py_compile backend/web.py` → OK.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The previous fix (setdefault) made `KeyError` non-fatal but did not
eliminate the underlying race: between `await websocket.accept()` and
the handler writing per-channel state, a concurrent broadcast may
add the just-accepted socket to its disconnected set and pop it from
ws_connections. The handler still completes, but its writes go to a
detached object that is never reattached; subsequent broadcasts
re-register a fresh state for the same socket and the duplicate
guard in setdefault suppression logic is per-socket, not per-channel,
so the first event after re-registration gets emitted unconditionally.
Wrap all reads, writes, and removals of `ws_connections` in a single
`asyncio.Lock` held only while touching the dict:
- `broadcast()` takes the lock to snapshot `(websocket, state)`
pairs, releases it before any `await`, and reacquires it to pop
dead sockets.
- `websocket_endpoint()` takes the lock to insert the new socket
immediately after `accept()`, and again after the initial-frame
`send_text` burst to record the per-channel state.
- Disconnect and exception handlers also pop under the lock.
`send_text` itself runs outside the lock, so concurrent broadcasts
still pipeline their frames in parallel (no global serialization of
network IO). The lock is only contended on dict mutation, which is
cheap.
Properties:
- No data races on ws_connections by construction.
- Slightly more lines than the setdefault variant but the mutation
surface is now declared explicitly via `async with self._ws_lock:`.
Diff stats: 35 insertions(+), 12 deletions(-) in `backend/web.py`.
Verified with `python3 -m py_compile`.
Self-standing fix, no API changes. Companion PRs:
- fix/ws-setdefault-and-broadcast (variant 2: simpler)
- fix/ws-writer-loop (variant 4: single-writer pattern)
All three fixes solve the same race. This one is the most explicit
about the mutation surface; setdefault is the smallest; writer-loop
removes any cross-task access entirely.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Symptom
Same as #58: production
brencher_brencher-backend(imageauto-74bf5aef-…)UI reconnect loop every ~5 s. Logs show
WebSocket /ws [accepted]followedby
KeyError: <starlette.websockets.WebSocket object …>atweb.py:175,closing the connection. Companion
brencher2(one stack, noSECONDARY_BRENCHER)shows 0 errors — same code is race-free without contention.
Root cause
Documented in #58.
ws_connectionsis mutated from two coroutines:broadcast()— invoked viaasyncio.run_coroutine_threadsafefrom theprocessing thread on every
onupdate(). With 7 profiles + theSecondaryManager callback, it runs ~10 times per ~100 s cycle.
websocket_endpoint()— FastAPI handler on the uvicorn loop. Betweenawait websocket.accept()and writing per-channel state, the handleryields control; a concurrent broadcast can pop the socket from the dict
before the handler records its initial state.
The setdefault variant (#58) keeps the broadcast write atomic but still
has a window where the handler writes to a state object that has already
been detached from
ws_connections. The next broadcast re-registers afresh state, with the duplicate-suppression guarantee now scoped per
re-registration rather than per channel.
Fix (this PR — variant
asyncio.Lock)Make every read, write, and removal of
ws_connectionsgo through asingle
asyncio.Lock. Hold the lock only while touching the dict —never across an
await. This turns three separate ad-hoc mutations intoone declared mutation surface (
async with self._ws_lock:).In
broadcast():async with self._ws_lock:snapshots(websocket, state)tuples,filters out the ones already up-to-date, and exits the block.
send_textruns outside the lock so concurrent broadcasts stillpipeline their frames in parallel (no global serialization of network IO).
async with self._ws_lock:pops the deadsockets in a single critical section.
In
websocket_endpoint():async with self._ws_lock:inserts the new socket immediately afteraccept().send_textburst, the handler takes the lockagain to read-back the (possibly-evicted-during-await) state. If the
state is gone (
getreturns None), the handler returns gracefully —the very next periodic broadcast will repopulate it.
In disconnect / exception handlers,
pop(websocket, None)is wrappedin
async with self._ws_lock:.Diff stats: 35 insertions(+), 12 deletions(-) in
backend/web.py.Compile-checked.
Companion PRs
fix/ws-setdefault-and-broadcast) — variant 2, smallest diff,atomic via
dict.setdefault. Less explicit about who coordinates,but cheaper to review and reason about.
fix/ws-writer-loop— variant 4, single-writer +asyncio.Queue.Removes all cross-task dict access by funnelling through one
consumer task. Larger change; useful if the lock ever shows up in
profiling.
Merge order I would recommend: #58 first (smallest blast radius if it
turns out to over-fix in some edge case). This PR keeps the explicit
locking pattern as the alternative for teams that prefer mutually
exclusive mutations over atomic dict primitives.
Validation
python3 -m py_compile backend/web.py→ OK.🤖 Generated with Claude Code
Pull request closed