fix(web): route all WS mutations through a single-writer task #60

Closed
hermes wants to merge 1 commits from hermes/brencher:fix/ws-writer-loop into main
Contributor

Symptom

Same as #58 and #59: production brencher_brencher-backend (image
auto-74bf5aef-…) shows WebSocket /ws [accepted] followed by
KeyError: <starlette.websockets.WebSocket object …> at web.py:175,
closing the connection every ~5 s. brencher2 (one stack, no
SECONDARY_BRENCHER) shows 0 errors.

Root cause

Documented in #58. Two coroutines mutate ws_connections —
broadcast() (via run_coroutine_threadsafe from the processing
thread) and websocket_endpoint() (on the uvicorn loop) — plus
the disconnect / exception cleanup paths. With 7 profiles plus the
SecondaryManager on_update callback, the broadcast path runs ~10
times per ~100 s cycle, and any of those broadcasts that observe
the just-accepted socket (because the cleanup pass added it to
disconnected) can pop it before websocket_endpoint writes
its initial state.

#58 (setdefault) closes the immediate KeyError symptom and #59
(asyncio.Lock) makes the mutation surface explicit. Both still have a
narrow race window where the handler writes to a state object that
has already been detached from ws_connections; the next broadcast
will then re-register a fresh entry, with the duplicate-suppression
guarantee scoped per re-registration rather than per channel.

Fix (this PR — variant single-writer)

Eliminate cross-coroutine access to ws_connections entirely by
funnelling every mutation through one task. WebApp owns an
asyncio.Queue populated with three kinds of items:

  • ("broadcast", message) — from broadcast().
  • ("register", websocket) — from websocket_endpoint() after
    accept(), replacing the direct ws_connections[websocket] = {}
    write.
  • ("unregister", websocket) — from WebSocketDisconnect /
    Exception cleanup, replacing the direct pop calls.

A single _writer_loop task consumes the queue and is the only
coroutine that ever reads or writes ws_connections. _deliver_frame
preserves the original dedupe-by-event semantics, simplified to a
single last_message per socket (a broadcast that produces the
exact same JSON payload as the previous one is dropped; otherwise
it is delivered) — matches the broadcast() original on the happy
path and is more predictable everywhere else.

The handler still flushes the initial version / branches /
environments frames synchronously to the client so the UI sees
the same opening burst it always did. WebApp.__init__ initialises
queue / task / ready-event before registering the FastAPI lifespan;
lifespan creates the writer task at startup and cancels it at
shutdown.

Diff stats: 82 insertions(+), 24 deletions(-) in backend/web.py.
Compile-checked.

Properties:

  • No cross-coroutine access to ws_connections — the original race
    is impossible, not just unlikely.
  • Producer code (broadcast*, emit_envs) is unchanged from the
    call site; only broadcast itself is reduced to
    await self._broadcast_queue.put(...).
  • Lifecycle is owned by FastAPI lifespan: the writer task is created
    at startup and cancelled at shutdown.

Caveats / follow-ups:

  • The writer task is single — if it dies the queue fills and
    broadcast* awaits indefinitely. A periodic ping from the UI
    could be used as a watchdog. Not introduced by this PR (was an
    implicit risk before too).
  • The handler's three initial send_text calls run before the
    register item is processed by the writer loop. The UI gets the
    same opening burst in the same order, so this is a no-op from its
    perspective.

Companion PRs

  • #58 (fix/ws-setdefault-and-broadcast) — variant 2, smallest
    diff. Atomic via dict.setdefault.
  • #59 (fix/ws-asyncio-lock) — variant 3, explicit asyncio.Lock
    on every read / write of ws_connections.
  • This PR — variant 4, removes cross-task access entirely.

Merge order I would recommend: #58 first. Promote to lock or
writer-loop only if profiling shows their cost; the writer-loop
variant is the heaviest but the most robust against future code
additions that try to touch ws_connections.

Validation

  • python3 -m py_compile backend/web.py → OK.

🤖 Generated with Claude Code

## Symptom Same as #58 and #59: production `brencher_brencher-backend` (image `auto-74bf5aef-…`) shows `WebSocket /ws [accepted]` followed by `KeyError: <starlette.websockets.WebSocket object …>` at `web.py:175`, closing the connection every ~5 s. `brencher2` (one stack, no `SECONDARY_BRENCHER`) shows 0 errors. ## Root cause Documented in #58. Two coroutines mutate `ws_connections` — `broadcast()` (via `run_coroutine_threadsafe` from the processing thread) and `websocket_endpoint()` (on the uvicorn loop) — plus the disconnect / exception cleanup paths. With 7 profiles plus the SecondaryManager on_update callback, the broadcast path runs ~10 times per ~100 s cycle, and any of those broadcasts that observe the just-accepted socket (because the cleanup pass added it to `disconnected`) can `pop` it before `websocket_endpoint` writes its initial state. `#58` (setdefault) closes the immediate `KeyError` symptom and `#59` (asyncio.Lock) makes the mutation surface explicit. Both still have a narrow race window where the handler writes to a state object that has already been detached from `ws_connections`; the next broadcast will then re-register a fresh entry, with the duplicate-suppression guarantee scoped per re-registration rather than per channel. ## Fix (this PR — variant `single-writer`) Eliminate cross-coroutine access to `ws_connections` entirely by funnelling every mutation through one task. `WebApp` owns an `asyncio.Queue` populated with three kinds of items: - `("broadcast", message)` — from `broadcast()`. - `("register", websocket)` — from `websocket_endpoint()` after `accept()`, replacing the direct `ws_connections[websocket] = {}` write. - `("unregister", websocket)` — from `WebSocketDisconnect` / `Exception` cleanup, replacing the direct `pop` calls. A single `_writer_loop` task consumes the queue and is the only coroutine that ever reads or writes `ws_connections`. `_deliver_frame` preserves the original dedupe-by-event semantics, simplified to a single `last_message` per socket (a broadcast that produces the exact same JSON payload as the previous one is dropped; otherwise it is delivered) — matches the broadcast() original on the happy path and is more predictable everywhere else. The handler still flushes the initial `version` / `branches` / `environments` frames synchronously to the client so the UI sees the same opening burst it always did. `WebApp.__init__` initialises queue / task / ready-event before registering the FastAPI lifespan; lifespan creates the writer task at startup and cancels it at shutdown. Diff stats: 82 insertions(+), 24 deletions(-) in `backend/web.py`. Compile-checked. Properties: - No cross-coroutine access to `ws_connections` — the original race is impossible, not just unlikely. - Producer code (`broadcast*`, `emit_envs`) is unchanged from the call site; only `broadcast` itself is reduced to `await self._broadcast_queue.put(...)`. - Lifecycle is owned by FastAPI lifespan: the writer task is created at startup and cancelled at shutdown. Caveats / follow-ups: - The writer task is single — if it dies the queue fills and broadcast* awaits indefinitely. A periodic `ping` from the UI could be used as a watchdog. Not introduced by this PR (was an implicit risk before too). - The handler's three initial `send_text` calls run before the `register` item is processed by the writer loop. The UI gets the same opening burst in the same order, so this is a no-op from its perspective. ## Companion PRs - #58 (`fix/ws-setdefault-and-broadcast`) — variant 2, smallest diff. Atomic via `dict.setdefault`. - #59 (`fix/ws-asyncio-lock`) — variant 3, explicit `asyncio.Lock` on every read / write of `ws_connections`. - This PR — variant 4, removes cross-task access entirely. Merge order I would recommend: #58 first. Promote to lock or writer-loop only if profiling shows their cost; the writer-loop variant is the heaviest but the most robust against future code additions that try to touch `ws_connections`. ## Validation - `python3 -m py_compile backend/web.py` → OK. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
hermes added 1 commit 2026-09-07 09:40:24 +02:00
fix(web): route all WS mutations through a single-writer task
Python package / build (3.12.3) (pull_request) Successful in 2m19s
Python package / build (3.13) (pull_request) Successful in 2m21s
Python package / build (3.14) (pull_request) Successful in 2m27s
5496388069
`ws_connections` was mutated by two coroutines — `broadcast()`
(launched via `run_coroutine_threadsafe` from the processing thread)
and `websocket_endpoint()` (the FastAPI handler) — plus the
`WebSocketDisconnect` and `Exception` cleanup paths. Under
contention (7 profiles + SecondaryManager on_update) the WS Error
logs from prod show the handler losing a `ws_connections[websocket]`
entry mid-flight, then crashing with KeyError when it tries to write
per-channel state.

Restructure so that `ws_connections` is touched from exactly one
coroutine — `_writer_loop`. It consumes an `asyncio.Queue` populated
by three kinds of items:
- `("broadcast", message)` from any `broadcast*` coroutine.
- `("register", websocket)` from `websocket_endpoint` after
  `accept()`, replacing the direct `ws_connections[websocket] = {}`
  write.
- `("unregister", websocket)` from disconnect / exception paths.

The handler still flushes the initial version / branches /
environments frames synchronously to the client (so the UI does
not see an empty window between accept and the first writer-loop
frame). The dedupe field is now a single `last_message` per
socket instead of per-event-key, which slightly changes the
suppression semantics: a broadcast that produces the exact same
JSON payload as the previous one (same event + same data) is
dropped, otherwise it is delivered. Behaviour matches the
broadcast() original on the happy path and is more predictable on
the rest.

Properties:
- No cross-coroutine access to `ws_connections` at all.
- Producer code (`broadcast`, `broadcast_*`, `emit_envs`) is
  unchanged from the call site; only the implementation of
  `broadcast` is reduced to `await self._broadcast_queue.put(...)`.
- Lifecycle is owned by FastAPI lifespan: the writer task is
  created at startup and cancelled at shutdown.
- Failure modes narrow:
  - If the writer task dies, the queue fills and broadcast* awaits
    indefinitely — needs a watchdog for production. Not introduced
    by this PR (was an implicit risk before too), noted for
    follow-up.
  - The handler still calls `send_text` synchronously for the
    initial three frames, so they can race with a register item
    already processed in `_writer_loop` — UI sees the initial set
    in the same order it always did.

Diff stats: 82 insertions(+), 24 deletions(-) in `backend/web.py`.
Compile-checked.

Companion PRs:
- #58 (setdefault) — variant 2, smallest diff.
- #59 (asyncio.Lock) — variant 3, explicit mutation surface.
- This PR (writer-loop) — variant 4, removes cross-task access
  entirely.

Merge order I would recommend: #58 first. Promote to lock or
writer-loop only if profiling shows their cost.

Self-standing fix. Optional follow-up: add an asyncio.Queue put timeout
and a watchdog on the writer task so a stuck send_text does not
stall the queue.
admin closed this pull request 2026-09-11 17:08:04 +02:00
admin deleted branch fix/ws-writer-loop 2026-09-11 17:08:08 +02:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/brencher#60