Same as #58 and #59: production brencher_brencher-backend (image auto-74bf5aef-…) shows WebSocket /ws [accepted] followed by KeyError: <starlette.websockets.WebSocket object …> at web.py:175,
closing the connection every ~5 s. brencher2 (one stack, no SECONDARY_BRENCHER) shows 0 errors.
Root cause
Documented in #58. Two coroutines mutate ws_connections — broadcast() (via run_coroutine_threadsafe from the processing
thread) and websocket_endpoint() (on the uvicorn loop) — plus
the disconnect / exception cleanup paths. With 7 profiles plus the
SecondaryManager on_update callback, the broadcast path runs ~10
times per ~100 s cycle, and any of those broadcasts that observe
the just-accepted socket (because the cleanup pass added it to disconnected) can pop it before websocket_endpoint writes
its initial state.
#58 (setdefault) closes the immediate KeyError symptom and #59
(asyncio.Lock) makes the mutation surface explicit. Both still have a
narrow race window where the handler writes to a state object that
has already been detached from ws_connections; the next broadcast
will then re-register a fresh entry, with the duplicate-suppression
guarantee scoped per re-registration rather than per channel.
Fix (this PR — variant single-writer)
Eliminate cross-coroutine access to ws_connections entirely by
funnelling every mutation through one task. WebApp owns an asyncio.Queue populated with three kinds of items:
("broadcast", message) — from broadcast().
("register", websocket) — from websocket_endpoint() after accept(), replacing the direct ws_connections[websocket] = {}
write.
("unregister", websocket) — from WebSocketDisconnect / Exception cleanup, replacing the direct pop calls.
A single _writer_loop task consumes the queue and is the only
coroutine that ever reads or writes ws_connections. _deliver_frame
preserves the original dedupe-by-event semantics, simplified to a
single last_message per socket (a broadcast that produces the
exact same JSON payload as the previous one is dropped; otherwise
it is delivered) — matches the broadcast() original on the happy
path and is more predictable everywhere else.
The handler still flushes the initial version / branches / environments frames synchronously to the client so the UI sees
the same opening burst it always did. WebApp.__init__ initialises
queue / task / ready-event before registering the FastAPI lifespan;
lifespan creates the writer task at startup and cancels it at
shutdown.
Diff stats: 82 insertions(+), 24 deletions(-) in backend/web.py.
Compile-checked.
Properties:
No cross-coroutine access to ws_connections — the original race
is impossible, not just unlikely.
Producer code (broadcast*, emit_envs) is unchanged from the
call site; only broadcast itself is reduced to await self._broadcast_queue.put(...).
Lifecycle is owned by FastAPI lifespan: the writer task is created
at startup and cancelled at shutdown.
Caveats / follow-ups:
The writer task is single — if it dies the queue fills and
broadcast* awaits indefinitely. A periodic ping from the UI
could be used as a watchdog. Not introduced by this PR (was an
implicit risk before too).
The handler's three initial send_text calls run before the register item is processed by the writer loop. The UI gets the
same opening burst in the same order, so this is a no-op from its
perspective.
Companion PRs
#58 (fix/ws-setdefault-and-broadcast) — variant 2, smallest
diff. Atomic via dict.setdefault.
#59 (fix/ws-asyncio-lock) — variant 3, explicit asyncio.Lock
on every read / write of ws_connections.
This PR — variant 4, removes cross-task access entirely.
Merge order I would recommend: #58 first. Promote to lock or
writer-loop only if profiling shows their cost; the writer-loop
variant is the heaviest but the most robust against future code
additions that try to touch ws_connections.
## Symptom
Same as #58 and #59: production `brencher_brencher-backend` (image
`auto-74bf5aef-…`) shows `WebSocket /ws [accepted]` followed by
`KeyError: <starlette.websockets.WebSocket object …>` at `web.py:175`,
closing the connection every ~5 s. `brencher2` (one stack, no
`SECONDARY_BRENCHER`) shows 0 errors.
## Root cause
Documented in #58. Two coroutines mutate `ws_connections` —
`broadcast()` (via `run_coroutine_threadsafe` from the processing
thread) and `websocket_endpoint()` (on the uvicorn loop) — plus
the disconnect / exception cleanup paths. With 7 profiles plus the
SecondaryManager on_update callback, the broadcast path runs ~10
times per ~100 s cycle, and any of those broadcasts that observe
the just-accepted socket (because the cleanup pass added it to
`disconnected`) can `pop` it before `websocket_endpoint` writes
its initial state.
`#58` (setdefault) closes the immediate `KeyError` symptom and `#59`
(asyncio.Lock) makes the mutation surface explicit. Both still have a
narrow race window where the handler writes to a state object that
has already been detached from `ws_connections`; the next broadcast
will then re-register a fresh entry, with the duplicate-suppression
guarantee scoped per re-registration rather than per channel.
## Fix (this PR — variant `single-writer`)
Eliminate cross-coroutine access to `ws_connections` entirely by
funnelling every mutation through one task. `WebApp` owns an
`asyncio.Queue` populated with three kinds of items:
- `("broadcast", message)` — from `broadcast()`.
- `("register", websocket)` — from `websocket_endpoint()` after
`accept()`, replacing the direct `ws_connections[websocket] = {}`
write.
- `("unregister", websocket)` — from `WebSocketDisconnect` /
`Exception` cleanup, replacing the direct `pop` calls.
A single `_writer_loop` task consumes the queue and is the only
coroutine that ever reads or writes `ws_connections`. `_deliver_frame`
preserves the original dedupe-by-event semantics, simplified to a
single `last_message` per socket (a broadcast that produces the
exact same JSON payload as the previous one is dropped; otherwise
it is delivered) — matches the broadcast() original on the happy
path and is more predictable everywhere else.
The handler still flushes the initial `version` / `branches` /
`environments` frames synchronously to the client so the UI sees
the same opening burst it always did. `WebApp.__init__` initialises
queue / task / ready-event before registering the FastAPI lifespan;
lifespan creates the writer task at startup and cancels it at
shutdown.
Diff stats: 82 insertions(+), 24 deletions(-) in `backend/web.py`.
Compile-checked.
Properties:
- No cross-coroutine access to `ws_connections` — the original race
is impossible, not just unlikely.
- Producer code (`broadcast*`, `emit_envs`) is unchanged from the
call site; only `broadcast` itself is reduced to
`await self._broadcast_queue.put(...)`.
- Lifecycle is owned by FastAPI lifespan: the writer task is created
at startup and cancelled at shutdown.
Caveats / follow-ups:
- The writer task is single — if it dies the queue fills and
broadcast* awaits indefinitely. A periodic `ping` from the UI
could be used as a watchdog. Not introduced by this PR (was an
implicit risk before too).
- The handler's three initial `send_text` calls run before the
`register` item is processed by the writer loop. The UI gets the
same opening burst in the same order, so this is a no-op from its
perspective.
## Companion PRs
- #58 (`fix/ws-setdefault-and-broadcast`) — variant 2, smallest
diff. Atomic via `dict.setdefault`.
- #59 (`fix/ws-asyncio-lock`) — variant 3, explicit `asyncio.Lock`
on every read / write of `ws_connections`.
- This PR — variant 4, removes cross-task access entirely.
Merge order I would recommend: #58 first. Promote to lock or
writer-loop only if profiling shows their cost; the writer-loop
variant is the heaviest but the most robust against future code
additions that try to touch `ws_connections`.
## Validation
- `python3 -m py_compile backend/web.py` → OK.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
`ws_connections` was mutated by two coroutines — `broadcast()`
(launched via `run_coroutine_threadsafe` from the processing thread)
and `websocket_endpoint()` (the FastAPI handler) — plus the
`WebSocketDisconnect` and `Exception` cleanup paths. Under
contention (7 profiles + SecondaryManager on_update) the WS Error
logs from prod show the handler losing a `ws_connections[websocket]`
entry mid-flight, then crashing with KeyError when it tries to write
per-channel state.
Restructure so that `ws_connections` is touched from exactly one
coroutine — `_writer_loop`. It consumes an `asyncio.Queue` populated
by three kinds of items:
- `("broadcast", message)` from any `broadcast*` coroutine.
- `("register", websocket)` from `websocket_endpoint` after
`accept()`, replacing the direct `ws_connections[websocket] = {}`
write.
- `("unregister", websocket)` from disconnect / exception paths.
The handler still flushes the initial version / branches /
environments frames synchronously to the client (so the UI does
not see an empty window between accept and the first writer-loop
frame). The dedupe field is now a single `last_message` per
socket instead of per-event-key, which slightly changes the
suppression semantics: a broadcast that produces the exact same
JSON payload as the previous one (same event + same data) is
dropped, otherwise it is delivered. Behaviour matches the
broadcast() original on the happy path and is more predictable on
the rest.
Properties:
- No cross-coroutine access to `ws_connections` at all.
- Producer code (`broadcast`, `broadcast_*`, `emit_envs`) is
unchanged from the call site; only the implementation of
`broadcast` is reduced to `await self._broadcast_queue.put(...)`.
- Lifecycle is owned by FastAPI lifespan: the writer task is
created at startup and cancelled at shutdown.
- Failure modes narrow:
- If the writer task dies, the queue fills and broadcast* awaits
indefinitely — needs a watchdog for production. Not introduced
by this PR (was an implicit risk before too), noted for
follow-up.
- The handler still calls `send_text` synchronously for the
initial three frames, so they can race with a register item
already processed in `_writer_loop` — UI sees the initial set
in the same order it always did.
Diff stats: 82 insertions(+), 24 deletions(-) in `backend/web.py`.
Compile-checked.
Companion PRs:
- #58 (setdefault) — variant 2, smallest diff.
- #59 (asyncio.Lock) — variant 3, explicit mutation surface.
- This PR (writer-loop) — variant 4, removes cross-task access
entirely.
Merge order I would recommend: #58 first. Promote to lock or
writer-loop only if profiling shows their cost.
Self-standing fix. Optional follow-up: add an asyncio.Queue put timeout
and a watchdog on the writer task so a stuck send_text does not
stall the queue.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Symptom
Same as #58 and #59: production
brencher_brencher-backend(imageauto-74bf5aef-…) showsWebSocket /ws [accepted]followed byKeyError: <starlette.websockets.WebSocket object …>atweb.py:175,closing the connection every ~5 s.
brencher2(one stack, noSECONDARY_BRENCHER) shows 0 errors.Root cause
Documented in #58. Two coroutines mutate
ws_connections—broadcast()(viarun_coroutine_threadsafefrom the processingthread) and
websocket_endpoint()(on the uvicorn loop) — plusthe disconnect / exception cleanup paths. With 7 profiles plus the
SecondaryManager on_update callback, the broadcast path runs ~10
times per ~100 s cycle, and any of those broadcasts that observe
the just-accepted socket (because the cleanup pass added it to
disconnected) canpopit beforewebsocket_endpointwritesits initial state.
#58(setdefault) closes the immediateKeyErrorsymptom and#59(asyncio.Lock) makes the mutation surface explicit. Both still have a
narrow race window where the handler writes to a state object that
has already been detached from
ws_connections; the next broadcastwill then re-register a fresh entry, with the duplicate-suppression
guarantee scoped per re-registration rather than per channel.
Fix (this PR — variant
single-writer)Eliminate cross-coroutine access to
ws_connectionsentirely byfunnelling every mutation through one task.
WebAppowns anasyncio.Queuepopulated with three kinds of items:("broadcast", message)— frombroadcast().("register", websocket)— fromwebsocket_endpoint()afteraccept(), replacing the directws_connections[websocket] = {}write.
("unregister", websocket)— fromWebSocketDisconnect/Exceptioncleanup, replacing the directpopcalls.A single
_writer_looptask consumes the queue and is the onlycoroutine that ever reads or writes
ws_connections._deliver_framepreserves the original dedupe-by-event semantics, simplified to a
single
last_messageper socket (a broadcast that produces theexact same JSON payload as the previous one is dropped; otherwise
it is delivered) — matches the broadcast() original on the happy
path and is more predictable everywhere else.
The handler still flushes the initial
version/branches/environmentsframes synchronously to the client so the UI seesthe same opening burst it always did.
WebApp.__init__initialisesqueue / task / ready-event before registering the FastAPI lifespan;
lifespan creates the writer task at startup and cancels it at
shutdown.
Diff stats: 82 insertions(+), 24 deletions(-) in
backend/web.py.Compile-checked.
Properties:
ws_connections— the original raceis impossible, not just unlikely.
broadcast*,emit_envs) is unchanged from thecall site; only
broadcastitself is reduced toawait self._broadcast_queue.put(...).at startup and cancelled at shutdown.
Caveats / follow-ups:
broadcast* awaits indefinitely. A periodic
pingfrom the UIcould be used as a watchdog. Not introduced by this PR (was an
implicit risk before too).
send_textcalls run before theregisteritem is processed by the writer loop. The UI gets thesame opening burst in the same order, so this is a no-op from its
perspective.
Companion PRs
fix/ws-setdefault-and-broadcast) — variant 2, smallestdiff. Atomic via
dict.setdefault.fix/ws-asyncio-lock) — variant 3, explicitasyncio.Lockon every read / write of
ws_connections.Merge order I would recommend: #58 first. Promote to lock or
writer-loop only if profiling shows their cost; the writer-loop
variant is the heaviest but the most robust against future code
additions that try to touch
ws_connections.Validation
python3 -m py_compile backend/web.py→ OK.🤖 Generated with Claude Code
`ws_connections` was mutated by two coroutines — `broadcast()` (launched via `run_coroutine_threadsafe` from the processing thread) and `websocket_endpoint()` (the FastAPI handler) — plus the `WebSocketDisconnect` and `Exception` cleanup paths. Under contention (7 profiles + SecondaryManager on_update) the WS Error logs from prod show the handler losing a `ws_connections[websocket]` entry mid-flight, then crashing with KeyError when it tries to write per-channel state. Restructure so that `ws_connections` is touched from exactly one coroutine — `_writer_loop`. It consumes an `asyncio.Queue` populated by three kinds of items: - `("broadcast", message)` from any `broadcast*` coroutine. - `("register", websocket)` from `websocket_endpoint` after `accept()`, replacing the direct `ws_connections[websocket] = {}` write. - `("unregister", websocket)` from disconnect / exception paths. The handler still flushes the initial version / branches / environments frames synchronously to the client (so the UI does not see an empty window between accept and the first writer-loop frame). The dedupe field is now a single `last_message` per socket instead of per-event-key, which slightly changes the suppression semantics: a broadcast that produces the exact same JSON payload as the previous one (same event + same data) is dropped, otherwise it is delivered. Behaviour matches the broadcast() original on the happy path and is more predictable on the rest. Properties: - No cross-coroutine access to `ws_connections` at all. - Producer code (`broadcast`, `broadcast_*`, `emit_envs`) is unchanged from the call site; only the implementation of `broadcast` is reduced to `await self._broadcast_queue.put(...)`. - Lifecycle is owned by FastAPI lifespan: the writer task is created at startup and cancelled at shutdown. - Failure modes narrow: - If the writer task dies, the queue fills and broadcast* awaits indefinitely — needs a watchdog for production. Not introduced by this PR (was an implicit risk before too), noted for follow-up. - The handler still calls `send_text` synchronously for the initial three frames, so they can race with a register item already processed in `_writer_loop` — UI sees the initial set in the same order it always did. Diff stats: 82 insertions(+), 24 deletions(-) in `backend/web.py`. Compile-checked. Companion PRs: - #58 (setdefault) — variant 2, smallest diff. - #59 (asyncio.Lock) — variant 3, explicit mutation surface. - This PR (writer-loop) — variant 4, removes cross-task access entirely. Merge order I would recommend: #58 first. Promote to lock or writer-loop only if profiling shows their cost. Self-standing fix. Optional follow-up: add an asyncio.Queue put timeout and a watchdog on the writer task so a stuck send_text does not stall the queue.Pull request closed