Race between websocket_endpoint and broadcast() in backend/web.py caused
intermittent 'KeyError: <WebSocket object>' errors every ~7s in
brencher-brencher-backend logs (counted 354+ in the last hour), forcing
the browser UI to reconnect and dropping log/sse updates.
Repro window:
websocket_endpoint:
175: self.ws_connections[websocket] = {}
await websocket.send_text(version_payload)
await websocket.send_text(branches_payload)
184: self.ws_connections[websocket]["branches"] = branches_payload # KeyError
broadcast() runs concurrently, sends to N sockets, awaits each, and
pops disconnected ones. The await yields control back to
websocket_endpoint which then tries to write a key that was already
popped (or — conversely — broadcast iterates over a snapshot while a
parallel endpoint adds a new key it later tries to mutate).
Fix:
- Add asyncio.Lock to WebApp.__init__ and acquire it for every mutation
of ws_connections (broadcast iteration + cleanup, websocket_endpoint
registration, WebSocketDisconnect cleanup).
- websocket_endpoint now registers the websocket with all three
payloads (version/branches/environments) in a single dict assignment
BEFORE the first send_text await, so there is no window where the key
exists but the inner dict is empty.
- broadcast() snapshots ws_connections.keys() via list() before iterating,
and runs send_text + dict write + cleanup under the same lock.
Tests (tests/test_web_ws_lock.py, 4 cases):
- test_websocket_endpoint_registers_atomically_under_lock: a single
endpoint connection with a slow send_text; concurrent broadcasts do
not cause KeyError or a missing-key dict.
- test_broadcast_and_concurrent_register_dont_crash: 20 endpoints + a
broadcaster all run concurrently; ws_connections must drain cleanly
and never raise.
- test_ws_lock_exists_and_is_asyncio_lock: guard against future
accidental removal.
- test_broadcast_skips_already_sent_payload: the de-dup check still
works after the lock change (regression guard).
Refs #42
Integration Tests for git.py
This directory contains integration tests for the CheckoutMerged and GitUnmerge classes from backend/steps/git.py.
Overview
The integration tests simulate real git operations in isolated temporary directories to ensure the git merge and unmerge functionality works correctly in various scenarios without affecting the host environment.
Test Structure
The tests are organized into two files:
test_git_integration.py- Main integration tests for CheckoutMerged and GitUnmergetest_remote_repo.py- Helper classes and utilities for creating test repositories and mock objects
Helper Classes
The test_remote_repo.py file provides:
RemoteRepoHelper- Manages creation and cleanup of temporary test repositoriesMockGitClone- Mock GitClone object for testingMockDockerSwarmCheck- Mock DockerSwarmCheck object with configurable version strings
Test Coverage
CheckoutMerged Class Tests
- test_checkout_merged_two_branches - Tests merging two branches successfully
- test_checkout_merged_three_branches - Tests merging three branches successfully
- test_checkout_merged_fast_forward - Tests merging with fast-forward (linear history)
- test_checkout_merged_conflicting_branches - Tests that conflicting branches fail gracefully
- test_checkout_merged_existing_auto_branch - Tests that existing auto branches are reused
- test_checkout_merged_empty_branches - Tests that empty branch lists are rejected
Combined CheckoutMerged and GitUnmerge Tests
- test_checkout_merged_and_unmerge_valid_version - Tests merging branches and then extracting branch information from version string
GitUnmerge-Specific Tests
- test_git_unmerge_invalid_version - Tests handling of invalid version formats
- test_git_unmerge_nonhead_commit - Tests behavior when version corresponds to non-HEAD commit in a branch ( expected to fail)
Test Improvements
Recent refactoring improvements:
- Centralized repository setup code in
RemoteRepoHelperclass - Single instances of
MockGitCloneandMockDockerSwarmCheckdefined in helper file - Enhanced assertions that check actual field values (not just "is not None")
- Merged GitUnmerge tests with CheckoutMerged tests where they share scenarios
Running the Tests
Option 1: Run in Docker (Recommended - Isolated Environment)
The tests can be run inside a Docker container to ensure complete isolation from the host system:
./run_integration_tests.sh
This script will:
- Build a Docker image with all necessary dependencies
- Run the integration tests inside a container
- Clean up after completion
Option 2: Run Locally
If you prefer to run tests locally (requires Python 3.12.3+ and git):
# Install dependencies
pip install -r requirements.txt
# Run tests
python -m pytest tests/test_git_integration.py -v
Option 3: Run Specific Tests
To run a specific test:
python -m pytest tests/test_git_integration.py::TestGitIntegration::test_checkout_merged_two_branches -v
Test Scenarios Simulated
Merge Scenarios
- 2 branches merge: Creates two divergent branches from a common ancestor and merges them
- 3 branches merge: Creates three divergent branches from a common ancestor and merges them
- Fast-forward merge: Tests merging when one branch is directly ahead of another
- Conflicting branches: Creates branches that modify the same file differently, expecting merge conflicts
Repository Simulation
Each test:
- Creates a temporary "remote" repository
- Creates a temporary "local" clone
- Simulates various branch scenarios
- Executes the merge/unmerge operations
- Verifies the results
- Cleans up all temporary files
CI Integration
These tests are integrated into the CI pipeline through the python-package.yml workflow, which runs pytest on every
push and pull request.
Dependencies
- Python 3.12.3+
- pytest
- gitpython
- git (system package)
All Python dependencies are specified in requirements.txt.