hermes e0ae7de150 fix(web): serialise ws_connections mutations under asyncio.Lock
Race between websocket_endpoint and broadcast() in backend/web.py caused
intermittent 'KeyError: <WebSocket object>' errors every ~7s in
brencher-brencher-backend logs (counted 354+ in the last hour), forcing
the browser UI to reconnect and dropping log/sse updates.

Repro window:
  websocket_endpoint:
    175: self.ws_connections[websocket] = {}
    await websocket.send_text(version_payload)
    await websocket.send_text(branches_payload)
    184: self.ws_connections[websocket]["branches"] = branches_payload  # KeyError

  broadcast() runs concurrently, sends to N sockets, awaits each, and
  pops disconnected ones. The await yields control back to
  websocket_endpoint which then tries to write a key that was already
  popped (or — conversely — broadcast iterates over a snapshot while a
  parallel endpoint adds a new key it later tries to mutate).

Fix:
- Add asyncio.Lock to WebApp.__init__ and acquire it for every mutation
  of ws_connections (broadcast iteration + cleanup, websocket_endpoint
  registration, WebSocketDisconnect cleanup).
- websocket_endpoint now registers the websocket with all three
  payloads (version/branches/environments) in a single dict assignment
  BEFORE the first send_text await, so there is no window where the key
  exists but the inner dict is empty.
- broadcast() snapshots ws_connections.keys() via list() before iterating,
  and runs send_text + dict write + cleanup under the same lock.

Tests (tests/test_web_ws_lock.py, 4 cases):
- test_websocket_endpoint_registers_atomically_under_lock: a single
  endpoint connection with a slow send_text; concurrent broadcasts do
  not cause KeyError or a missing-key dict.
- test_broadcast_and_concurrent_register_dont_crash: 20 endpoints + a
  broadcaster all run concurrently; ws_connections must drain cleanly
  and never raise.
- test_ws_lock_exists_and_is_asyncio_lock: guard against future
  accidental removal.
- test_broadcast_skips_already_sent_payload: the de-dup check still
  works after the lock change (regression guard).

Refs #42
2026-09-05 17:52:44 +00:00
2026-03-19 17:01:23 +01:00
2026-08-12 21:09:07 +02:00
2026-03-19 19:34:35 +01:00
2026-08-12 21:09:07 +02:00
2026-05-09 17:31:50 +02:00
2026-03-19 17:01:23 +01:00
2026-01-20 22:49:40 +01:00
2026-03-19 17:01:23 +01:00
2026-03-20 14:00:56 +01:00
2026-03-19 17:01:23 +01:00

Technical description

Use language python.

Locally run in venv create with uv tool. Use venv with uv for Python environment management and dependency installation. Keep requirements.txt updated.

Backend server flask.

UI - use plain javascript

App should be delivered as docker image.

Generate docker compose file to deploy to docker swarm.

Local start

python ./backend/app.py brencher_local1:docker_swarm_check_by_version noweb

Prod recovery

uv venv --python 3.12.3 .venv

uv pip install -r requirements.txt

python ./backend/app.py brencher:docker_swarm_check_by_version noweb

Update docker secret

docker service scale brencher_brencher-backend=0 brencher2_brencher-backend=0 docker service update --secret-rm brencher-secrets brencher_brencher-backend docker service update --secret-rm brencher-secrets brencher2_brencher-backend docker secret rm brencher-secrets

printf "GIT_USERNAME=git\nGIT_PASSWORD=TODO\n" |
docker secret create brencher-secrets -

docker service update --secret-add brencher-secrets brencher_brencher-backend docker service update --secret-add brencher-secrets brencher2_brencher-backend docker service scale brencher_brencher-backend=1 brencher2_brencher-backend=1

S
Description
No description provided
Readme
1.2 MiB
Languages
Python 84.3%
JavaScript 12.1%
CSS 2.1%
HTML 0.9%
Dockerfile 0.6%