Race between websocket_endpoint and broadcast() in backend/web.py caused
intermittent 'KeyError: <WebSocket object>' errors every ~7s in
brencher-brencher-backend logs (counted 354+ in the last hour), forcing
the browser UI to reconnect and dropping log/sse updates.
Repro window:
websocket_endpoint:
175: self.ws_connections[websocket] = {}
await websocket.send_text(version_payload)
await websocket.send_text(branches_payload)
184: self.ws_connections[websocket]["branches"] = branches_payload # KeyError
broadcast() runs concurrently, sends to N sockets, awaits each, and
pops disconnected ones. The await yields control back to
websocket_endpoint which then tries to write a key that was already
popped (or — conversely — broadcast iterates over a snapshot while a
parallel endpoint adds a new key it later tries to mutate).
Fix:
- Add asyncio.Lock to WebApp.__init__ and acquire it for every mutation
of ws_connections (broadcast iteration + cleanup, websocket_endpoint
registration, WebSocketDisconnect cleanup).
- websocket_endpoint now registers the websocket with all three
payloads (version/branches/environments) in a single dict assignment
BEFORE the first send_text await, so there is no window where the key
exists but the inner dict is empty.
- broadcast() snapshots ws_connections.keys() via list() before iterating,
and runs send_text + dict write + cleanup under the same lock.
Tests (tests/test_web_ws_lock.py, 4 cases):
- test_websocket_endpoint_registers_atomically_under_lock: a single
endpoint connection with a slow send_text; concurrent broadcasts do
not cause KeyError or a missing-key dict.
- test_broadcast_and_concurrent_register_dont_crash: 20 endpoints + a
broadcaster all run concurrently; ws_connections must drain cleanly
and never raise.
- test_ws_lock_exists_and_is_asyncio_lock: guard against future
accidental removal.
- test_broadcast_skips_already_sent_payload: the de-dup check still
works after the lock change (regression guard).
Refs #42