Race between websocket_endpoint and broadcast() in backend/web.py caused
intermittent 'KeyError: <WebSocket object>' errors every ~7s in
brencher-brencher-backend logs (counted 354+ in the last hour), forcing
the browser UI to reconnect and dropping log/sse updates.
Repro window:
websocket_endpoint:
175: self.ws_connections[websocket] = {}
await websocket.send_text(version_payload)
await websocket.send_text(branches_payload)
184: self.ws_connections[websocket]["branches"] = branches_payload # KeyError
broadcast() runs concurrently, sends to N sockets, awaits each, and
pops disconnected ones. The await yields control back to
websocket_endpoint which then tries to write a key that was already
popped (or — conversely — broadcast iterates over a snapshot while a
parallel endpoint adds a new key it later tries to mutate).
Fix:
- Add asyncio.Lock to WebApp.__init__ and acquire it for every mutation
of ws_connections (broadcast iteration + cleanup, websocket_endpoint
registration, WebSocketDisconnect cleanup).
- websocket_endpoint now registers the websocket with all three
payloads (version/branches/environments) in a single dict assignment
BEFORE the first send_text await, so there is no window where the key
exists but the inner dict is empty.
- broadcast() snapshots ws_connections.keys() via list() before iterating,
and runs send_text + dict write + cleanup under the same lock.
Tests (tests/test_web_ws_lock.py, 4 cases):
- test_websocket_endpoint_registers_atomically_under_lock: a single
endpoint connection with a slow send_text; concurrent broadcasts do
not cause KeyError or a missing-key dict.
- test_broadcast_and_concurrent_register_dont_crash: 20 endpoints + a
broadcaster all run concurrently; ws_connections must drain cleanly
and never raise.
- test_ws_lock_exists_and_is_asyncio_lock: guard against future
accidental removal.
- test_broadcast_skips_already_sent_payload: the de-dup check still
works after the lock change (regression guard).
Refs #42
Technical description
Use language python.
Locally run in venv create with uv tool. Use venv with uv for Python environment management and dependency installation. Keep requirements.txt updated.
Backend server flask.
UI - use plain javascript
App should be delivered as docker image.
Generate docker compose file to deploy to docker swarm.
Local start
python ./backend/app.py brencher_local1:docker_swarm_check_by_version noweb
Prod recovery
uv venv --python 3.12.3 .venv
uv pip install -r requirements.txt
python ./backend/app.py brencher:docker_swarm_check_by_version noweb
Update docker secret
docker service scale brencher_brencher-backend=0 brencher2_brencher-backend=0 docker service update --secret-rm brencher-secrets brencher_brencher-backend docker service update --secret-rm brencher-secrets brencher2_brencher-backend docker secret rm brencher-secrets
printf "GIT_USERNAME=git\nGIT_PASSWORD=TODO\n" |
docker secret create brencher-secrets -
docker service update --secret-add brencher-secrets brencher_brencher-backend docker service update --secret-add brencher-secrets brencher2_brencher-backend docker service scale brencher_brencher-backend=1 brencher2_brencher-backend=1