Compare commits

...
Author SHA1 Message Date
hermes e0ae7de150 fix(web): serialise ws_connections mutations under asyncio.Lock
Race between websocket_endpoint and broadcast() in backend/web.py caused
intermittent 'KeyError: <WebSocket object>' errors every ~7s in
brencher-brencher-backend logs (counted 354+ in the last hour), forcing
the browser UI to reconnect and dropping log/sse updates.

Repro window:
  websocket_endpoint:
    175: self.ws_connections[websocket] = {}
    await websocket.send_text(version_payload)
    await websocket.send_text(branches_payload)
    184: self.ws_connections[websocket]["branches"] = branches_payload  # KeyError

  broadcast() runs concurrently, sends to N sockets, awaits each, and
  pops disconnected ones. The await yields control back to
  websocket_endpoint which then tries to write a key that was already
  popped (or — conversely — broadcast iterates over a snapshot while a
  parallel endpoint adds a new key it later tries to mutate).

Fix:
- Add asyncio.Lock to WebApp.__init__ and acquire it for every mutation
  of ws_connections (broadcast iteration + cleanup, websocket_endpoint
  registration, WebSocketDisconnect cleanup).
- websocket_endpoint now registers the websocket with all three
  payloads (version/branches/environments) in a single dict assignment
  BEFORE the first send_text await, so there is no window where the key
  exists but the inner dict is empty.
- broadcast() snapshots ws_connections.keys() via list() before iterating,
  and runs send_text + dict write + cleanup under the same lock.

Tests (tests/test_web_ws_lock.py, 4 cases):
- test_websocket_endpoint_registers_atomically_under_lock: a single
  endpoint connection with a slow send_text; concurrent broadcasts do
  not cause KeyError or a missing-key dict.
- test_broadcast_and_concurrent_register_dont_crash: 20 endpoints + a
  broadcaster all run concurrently; ws_connections must drain cleanly
  and never raise.
- test_ws_lock_exists_and_is_asyncio_lock: guard against future
  accidental removal.
- test_broadcast_skips_already_sent_payload: the de-dup check still
  works after the lock change (regression guard).

Refs #42
2026-09-05 17:52:44 +00:00
hermes 2556ac76f7 Merge remote-tracking branch 'admin/main' into HEAD
# Conflicts:
#	backend/web.py
#	tests/test_dry_run_playwright.py
2026-09-05 17:34:17 +00:00
hermesandadmin 99f8b73654 feat(git): per-remote branch prefix filtering for multi-remote GitClone (#31)
## Summary

`GitClone`'s top-level `branchNamePrefix` argument was silently ignored when constructed with the explicit `remotes=[...]` form (verified in `backend/steps/git.py:106`):

```python
if remote_name == "origin" and self.branchNamePrefix != "":
    # apply refspec
```

The prefix only fires when a remote is named `"origin"`. The `remotes=` form passes names through verbatim, so configs like the open PR #28 set the prefix but never apply it — all branches from all remotes are fetched.

This PR extends the API: each remote entry now optionally accepts a `prefix` field. When set, the fetch refspec for that remote becomes scoped to `<prefix>/*` branches. Backwards-compatible — remotes without `prefix` keep the existing behavior, and `url=` shorthand with `branchNamePrefix` still works.

## API

```python
clone = GitClone(remotes=[
    {"name": "github",       "url": "...", "prefix": "ansible"},
    {"name": "gitea_hermes", "url": "...", "prefix": "ansible"},
    {"name": "gitea_admin",  "url": "...", "prefix": "ansible"},
])
```

Each remote is independent — some can have `prefix`, others not.

## Changes

### `backend/steps/git.py`

- `__init__` (`remotes=` branch): validate `prefix` (string if set, reject non-string with `ValueError`), persist into the normalized entry. Empty string / None → no prefix applied (existing behavior).
- `progress()` fetch loop: replace the single `if remote_name == "origin"` check with per-entry logic. Effective prefix = `entry["prefix"] or (self.branchNamePrefix if remote_name == "origin" else "")`. Apply refspec per remote when effective prefix is non-empty.

### `tests/test_git_integration.py`

3 new tests in `TestGitIntegration`:

- `test_git_clone_remotes_per_remote_prefix` — both remotes prefixed; only `ansible/*` branches visible, `master` is filtered out.
- `test_git_clone_remotes_mixed_prefix` — one remote prefixed, one not; each behaves independently.
- `test_git_clone_remotes_prefix_validation` — `prefix=123` / `prefix=["ansible"]` raise `ValueError`; `prefix=""` and missing-`prefix` are accepted.

Existing `test_git_clone_named_remotes` continues to pass (verified — no behavior change for unprefixed remotes).

## Test results

```
29 passed in ~13s
```

(18 in `test_git_integration.py`, 11 in `test_dummy.py`/`test_caching_step.py`/`test_cached_env.py`.)

## Migration path

After merge:

- Open migration PRs (#26 gmail_mcp, #28 hermes, planned #19 gitea.py) — switch from `branchNamePrefix="ansible"` to `"prefix": "ansible"` on each remote.
- `hermes_infra.py` after PR #8 already drops `branchNamePrefix`, no change needed.
- Existing `url=` configs (immich, gmail_mcp-old, hermes-old, registry, gitea-old) — unaffected.

## Unblocks

PR #28 — once this lands, hermes.py can drop the (currently misleading) top-level `branchNamePrefix="ansible"` and add `"prefix": "ansible"` per remote.

Refs #30
Blocks #28
Part of #18 (parent migration ticket)

---------

Co-authored-by: admin <rudolfss13@gmail.com>
Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/31
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-05 19:20:44 +02:00
hermesandadmin 272504eab3 feat(configs): point gmail_mcp at gitea admin/hermes mcps repos (#26)
## Summary

Updates `backend/configs/gmail_mcp.py` to point at the new `mcps` stack mirrors on gitea instead of the legacy `uber_backup` repo.

Remotes now:

- `gitea_hermes` → `https://gitea.rudolf.keenetic.link/hermes/mcps`
- `gitea_admin`  → `https://gitea.rudolf.keenetic.link/admin/mcps`

Uses the `remotes=[{name, url}, …]` form introduced by #3.

## What changed vs the original #26 body

- **Repo switched**: `uber_backup` → `mcps` (the gmail + telegram MCP server repo, now mirrored on gitea under both `admin/` and `hermes/`).
- **GitHub remote dropped**: matches the pattern in `hermes_infra.py` after #8 and `docker_socket_proxy.py` after #23. The brencher container cannot reach `github.com` (squid returns HTTP 403 on CONNECT tunnel), so keeping a `github` remote would hard-fail the clone and trap brencher in the same retry loop that blocked #23 on `admin/reverseproxy` before its mirror was created.
- **`branchNamePrefix="immoscout"` dropped**: the prefix is only applied to remotes named `origin` (see `backend/steps/git.py` ~L104); neither remote qualifies, so the field is a silent no-op. Matches the cleanup done in `hermes_infra.py` / `brencher.py`.

## Diff (vs `main`)

```diff
-clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="immoscout")
+clone = GitClone(remotes=[
+    {"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/mcps"},
+    {"name": "gitea_admin",  "url": "https://gitea.rudolf.keenetic.link/admin/mcps"},
+])
```

## Prerequisites (already done)

Both target mirrors exist on gitea:

- `admin/mcps` — already present (primary, not a fork).
- `hermes/mcps` — forked in this session (id=10, fork of `admin/mcps`).

## Test plan

1. From the brencher state file for `gmail_mcp`, register `gitea_hermes/feat/migrate-gmail-mcp-to-multi-remote` as the desired branch.
2. Watch brencher logs: `Cloning repositories [...]` should succeed for both `gitea_hermes` and `gitea_admin`.
3. Confirm `gmail_mcp` stack redeploys against the new repo.

Refs #20
Part of #18

---------

Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/26
Co-authored-by: brencher_bot (hermes) <rudolfss13+hermes@gmail.com>
2026-09-05 19:20:16 +02:00
hermesandhermes fa9f8791bd feat(configs): point immich at gitea admin/hermes ansible repos (#12)
## Summary
Migrates `backend/configs/immich.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/immich.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git"[, branchNamePrefix=...])`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why migrate
- The old URL never resolved (`uber_backup.git` doesn't exist on gitea anyway) — so the deploy path was broken even though the file looked correct.
- `github.com` returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (commit `653aee9`).

## Scope decision: only `immich.py`
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) are being migrated in **separate PRs**, one per file. Each PR has its own branch + verification.

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/immich.py` — OK
- Real import via `from configs import immich`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''`
- pytest: **26/26** passed on the no-docker / no-playwright suite
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible`.
- end-to-end `docker stack deploy` — needs a real swarm.

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of this environment will clone from gitea instead of the unreachable github URL.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/12
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-05 19:19:47 +02:00
hermesandhermes 694c28109d feat(configs): point hermes at gitea admin/hermes ansible repos (#11)
## Summary
Migrates `backend/configs/hermes.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/hermes.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git"[, branchNamePrefix=...])`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why migrate
- The old URL never resolved (`uber_backup.git` doesn't exist on gitea anyway) — so the deploy path was broken even though the file looked correct.
- `github.com` returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (commit `653aee9`).

## Scope decision: only `hermes.py`
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) are being migrated in **separate PRs**, one per file. Each PR has its own branch + verification.

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/hermes.py` — OK
- Real import via `from configs import hermes`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''`
- pytest: **26/26** passed on the no-docker / no-playwright suite
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible`.
- end-to-end `docker stack deploy` — needs a real swarm.

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of this environment will clone from gitea instead of the unreachable github URL.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/11
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-05 19:19:22 +02:00
copilot-swe-agent[bot]andrudolf1 0e43c0fd13 harden playwright test config and assertions
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/d019d010-3ca0-4a73-8826-620eef58edab

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 20:08:12 +00:00
copilot-swe-agent[bot]andrudolf1 ab01fca841 stabilize dry-run playwright integration
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/d019d010-3ca0-4a73-8826-620eef58edab

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 20:06:11 +00:00
copilot-swe-agent[bot]andrudolf1 b2782ee220 clarify async taskgroup behavior
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:36:56 +00:00
copilot-swe-agent[bot]andrudolf1 1acea85934 add async shutdown cleanup and review fixes
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:35:59 +00:00
copilot-swe-agent[bot]andrudolf1 6f3dd6ebd8 refactor app concurrency to asyncio taskgroup
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:34:46 +00:00
copilot-swe-agent[bot] efe49119a3 Initial plan 2026-05-19 19:02:30 +00:00
13 changed files with 636 additions and 134 deletions
+3 -9
View File
@@ -1,14 +1,10 @@
FROM nginx:1.27-alpine
COPY .git /tmp/repo/.git
# Configure nginx to listen on 5001 to match container port expectations.
RUN sed -i 's/listen 80;/listen 5001;/' /etc/nginx/conf.d/default.conf
# Build a custom landing page directly in the image and inject git revision.
# Build a custom landing page directly in the image.
RUN set -eux; \
apk add --no-cache --virtual .git-deps git; \
GIT_REVISION="$(git --git-dir=/tmp/repo/.git rev-parse --short HEAD 2>/dev/null || echo unknown)"; \
{ \
echo '<!doctype html>'; \
echo '<html lang="en">'; \
@@ -21,11 +17,9 @@ RUN set -eux; \
echo ' <h1>Nginx Is Running</h1>'; \
echo ' <p>Custom landing page for the Brencher test app container.</p>'; \
echo ' <p>Port: 5001</p>'; \
echo " <p>Git revision: ${GIT_REVISION}</p>"; \
echo ' <p>Git revision: unknown</p>'; \
echo '</body>'; \
echo '</html>'; \
} > /usr/share/nginx/html/index.html; \
rm -rf /tmp/repo/.git; \
apk del .git-deps
} > /usr/share/nginx/html/index.html
EXPOSE 5001
+72 -21
View File
@@ -1,8 +1,8 @@
import asyncio
import logging
import os
import signal
import sys
import threading
import traceback
from typing import Any, Callable, Dict, List, Optional
@@ -27,9 +27,12 @@ class App:
def __init__(self, environments: Dict[str, Environment]) -> None:
self.environments: Dict[str, Environment] = {id: wrap_in_cached(e) for id, e in environments.items()}
self.state_lock = threading.Lock()
self.environment_update_event = threading.Event()
self.environment_update_event = asyncio.Event()
self.environment_update_version = 0
self.emit_callback: Optional[Callable[[], None]] = None
self.reset_requested = False
self.shutdown_event = asyncio.Event()
self.web_app: Any = None
def get_local_envs_to_emit(self) -> Dict[str, Dict[str, Any]]:
env_dtos: Dict[str, Dict[str, Any]] = {}
@@ -88,35 +91,83 @@ class App:
return branches
def processing_thread(self) -> None:
while True:
async def processing_loop(self) -> None:
while not self.shutdown_event.is_set():
import processing
emit = self.emit_callback or (lambda: None)
update_version = self.environment_update_version
with self.state_lock:
logger.info(f"Processing")
if processing.process_all_jobs(list(self.environments.values()), emit):
self.environment_update_event.wait(timeout=1 * 5)
else:
self.environment_update_event.wait(timeout=1 * 60)
self.environment_update_event.clear()
logger.info("Processing")
if self.reset_requested:
await asyncio.to_thread(processing.reset_caches, list(self.environments.values()))
self.reset_requested = False
has_error = await asyncio.to_thread(processing.process_all_jobs, list(self.environments.values()), emit)
if self.shutdown_event.is_set():
return
# Skip the wait when a newer update arrived while the current processing pass was running.
if self.environment_update_version != update_version:
continue
timeout = 5 if has_error else 60
try:
await asyncio.wait_for(self.environment_update_event.wait(), timeout=timeout)
except asyncio.TimeoutError:
pass
self.environment_update_event.clear()
def run(self) -> None:
processing = threading.Thread(target=self.processing_thread)
processing.daemon = True
processing.start()
def notify_environment_update(self) -> None:
self.environment_update_version += 1
self.environment_update_event.set()
def request_reset(self) -> None:
self.reset_requested = True
def stop(self) -> None:
self.shutdown_event.set()
self.notify_environment_update()
if self.web_app is not None:
self.web_app.stop()
async def runAsync(self) -> None:
await self.processing_loop()
async def runHeadlessAsync(self) -> None:
await self.processing_loop()
def runHeadless(self) -> None:
"""Run the processing loop on the current thread; blocks forever."""
self.processing_thread()
"""Run the processing loop until cancelled; blocks forever."""
asyncio.run(self.runHeadlessAsync())
def runWeb(self, port: int) -> None:
async def _run_web_server(self, web_app: Any) -> None:
"""Raise a sentinel exception when the server stops so TaskGroup cancels processing."""
await web_app.start_async()
raise _WebServerStopped()
async def runWebAsync(self, port: int) -> None:
import web
web_app = web.WebApp(core=self, port=port)
self.web_app = web_app
self.emit_callback = web_app.emit_envs
self.run()
web_app.start()
try:
# Keep processing and web serving coupled so shutdown or fatal failure tears down the whole app.
async with asyncio.TaskGroup() as task_group:
task_group.create_task(self.processing_loop())
task_group.create_task(self._run_web_server(web_app))
except* _WebServerStopped:
pass
finally:
self.emit_callback = None
self.web_app = None
def runWeb(self, port: int) -> None:
asyncio.run(self.runWebAsync(port))
def run(self) -> None:
asyncio.run(self.runAsync())
class _WebServerStopped(Exception):
pass
if __name__ == '__main__':
cli_env_ids_list = sys.argv[1:]
+6 -3
View File
@@ -4,7 +4,10 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="immoscout")
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/mcps"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/mcps"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="gmail_mcp",
@@ -23,7 +26,7 @@ checkoutMerged = CheckoutMerged(clone,
buildDocker = DockerComposeBuild(clone,
docker_repo_username="",
docker_repo_password="",
docker_compose_path="gmail-mcp-server/docker-compose.yml",
docker_compose_path="docker-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
envs=lambda: {
@@ -39,7 +42,7 @@ deployDocker = DockerSwarmDeploy(
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="gmail_mcp",
docker_compose_path="gmail-mcp-server/docker-compose.yml",
docker_compose_path="docker-compose.yml",
)
checkPing1 = UrlCheck(
+22 -1
View File
@@ -4,7 +4,28 @@ from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="hermes",
+22 -1
View File
@@ -4,7 +4,28 @@ from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="immich",
+11 -4
View File
@@ -48,8 +48,13 @@ class GitClone(AbstractStep[str]):
raise ValueError(f"Remote {name!r} requires a non-empty 'url' (string)")
if name in seen:
raise ValueError(f"Duplicate remote name: {name!r}")
prefix_value = entry.get("prefix")
if prefix_value is not None and not isinstance(prefix_value, str):
raise ValueError(
f"Remote {name!r} 'prefix' must be a string when set, got {type(prefix_value).__name__}"
)
seen.add(name)
normalized.append({"name": name, "url": url_value})
normalized.append({"name": name, "url": url_value, "prefix": prefix_value or ""})
elif url is not None:
if isinstance(url, str):
raw = [url]
@@ -103,10 +108,12 @@ class GitClone(AbstractStep[str]):
remote.set_url(auth_url)
else:
remote = repo.create_remote(remote_name, auth_url)
if remote_name == "origin" and self.branchNamePrefix != "":
prefix = entry.get("prefix", "")
effective_prefix = prefix or (self.branchNamePrefix if remote_name == "origin" else "")
if effective_prefix:
with repo.config_writer() as config:
config.set_value('remote "origin"', "fetch",
f"+refs/heads/{self.branchNamePrefix}/*:refs/remotes/origin/{self.branchNamePrefix}/*")
config.set_value(f'remote "{remote_name}"', "fetch",
f"+refs/heads/{effective_prefix}/*:refs/remotes/{remote_name}/{effective_prefix}/*")
result = remote.fetch(prune=True)
if not result or any(fetch_info.flags & fetch_info.ERROR for fetch_info in result):
raise BaseException(f"Failed to fetch updates for {repository_url}")
+68 -33
View File
@@ -10,12 +10,10 @@ from fastapi import FastAPI, Response, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse
from app import App
from enironment import SharedStateConflictError, get_step
from app import App
from utils import custom_json_dumps
from processing import reset_caches
from secondary import SecondaryManager
from steps.step import CachingStep
logger = logging.getLogger(__name__)
@@ -47,12 +45,14 @@ class WebApp:
self.port = port
self.ws_connections: Dict[WebSocket, Dict[str, Any]] = {}
self.ws_lock = asyncio.Lock()
self._event_loop: Optional[asyncio.AbstractEventLoop] = None
self._server: Any = None
self.secondaryManager: Optional[SecondaryManager] = None
@asynccontextmanager
async def lifespan(fastapi_app: FastAPI) -> AsyncIterator[None]:
self._event_loop = asyncio.get_event_loop()
self._event_loop = asyncio.get_running_loop()
self.secondaryManager = SecondaryManager(
urls=os.getenv("SECONDARY_BRENCHER", ""),
on_update=lambda: self.broadcast_all(),
@@ -94,10 +94,20 @@ class WebApp:
merged = _merge_dicts(merged, connector.branches)
return merged
# --- Async scheduling from sync threads ---
# --- Async scheduling from sync callbacks ---
def _schedule_async(self, coro: Any) -> None:
if self._event_loop is not None and self._event_loop.is_running():
if self._event_loop is None or not self._event_loop.is_running():
logger.warning("Dropping scheduled coroutine because the web event loop is not running")
coro.close()
return
try:
running_loop = asyncio.get_running_loop()
except RuntimeError:
running_loop = None
if running_loop is self._event_loop:
self._event_loop.create_task(coro)
else:
asyncio.run_coroutine_threadsafe(coro, self._event_loop)
# --- Broadcasting ---
@@ -107,18 +117,19 @@ class WebApp:
disconnected = set()
message = custom_json_dumps({event: data})
for websocket in self.ws_connections.keys():
if self.ws_connections[websocket].get(event) == message:
continue
try:
await websocket.send_text(message)
self.ws_connections[websocket][event] = message
except Exception as e:
logger.error(f"Error sending to websocket: {e}")
disconnected.add(websocket)
async with self.ws_lock:
for websocket in list(self.ws_connections.keys()):
if self.ws_connections[websocket].get(event) == message:
continue
try:
await websocket.send_text(message)
self.ws_connections[websocket][event] = message
except Exception as e:
logger.error(f"Error sending to websocket: {e}")
disconnected.add(websocket)
for ws in disconnected:
self.ws_connections.pop(ws, None)
for ws in disconnected:
self.ws_connections.pop(ws, None)
async def broadcast_all(self) -> None:
await self.broadcast("branches", self.get_global_branches_to_emit())
@@ -135,7 +146,7 @@ class WebApp:
await self.broadcast("error", data)
def emit_envs(self) -> None:
"""Sync callback used by App's processing thread to push updates to clients."""
"""Sync callback used by App's processing loop to push updates to clients."""
self._schedule_async(self.broadcast_branches(self.get_global_branches_to_emit()))
self._schedule_async(self.broadcast_environments(self.get_global_envs_to_emit()))
@@ -163,18 +174,26 @@ class WebApp:
async def websocket_endpoint(self, websocket: WebSocket) -> None:
await websocket.accept()
self.ws_connections[websocket] = {}
# Compute payloads first so the dict registration below is a single
# atomic update under ws_lock — avoids the window where the key
# exists but the inner dict is empty, which races with broadcast()
# cleanup and crashes with KeyError at self.ws_connections[ws][...] = ...
branches_payload = custom_json_dumps({"branches": self.get_global_branches_to_emit()})
envs_payload = custom_json_dumps({"environments": self.get_global_envs_to_emit()})
version_payload = custom_json_dumps({"version": self.get_version()})
async with self.ws_lock:
self.ws_connections[websocket] = {
"version": version_payload,
"branches": branches_payload,
"environments": envs_payload,
}
try:
# Send initial state on connect and record it so duplicate broadcasts are suppressed
branches_payload = custom_json_dumps({"branches": self.get_global_branches_to_emit()})
envs_payload = custom_json_dumps({"environments": self.get_global_envs_to_emit()})
version_payload = custom_json_dumps({"version": self.get_version()})
await websocket.send_text(version_payload)
await websocket.send_text(branches_payload)
self.ws_connections[websocket]["branches"] = branches_payload
await websocket.send_text(envs_payload)
self.ws_connections[websocket]["environments"] = envs_payload
while True:
data = await websocket.receive_text()
@@ -187,7 +206,7 @@ class WebApp:
await self.secondaryManager.send({"update": update_data})
id = update_data.get('id', '')
if id == '':
reset_caches(list(self.core.environments.values()))
self.core.request_reset()
elif id not in self.core.environments.keys() and id not in {j for it in self.secondaryManager or [] for j in it.environments.keys()}:
logger.warning(f"Received update for unknown environment id {id}")
continue
@@ -218,22 +237,38 @@ class WebApp:
continue
if env:
p = get_step(env.pipeline, type(env.state))
if isinstance(p, CachingStep):
p.reset()
self.core.request_reset()
logger.info(f"Updated environment {env.id} branches to {update_data.get('branches')}, dry={update_data.get('dry')}")
self.core.environment_update_event.set()
self.core.notify_environment_update()
await self.broadcast_environments(self.get_global_envs_to_emit())
except WebSocketDisconnect:
self.ws_connections.pop(websocket, None)
async with self.ws_lock:
self.ws_connections.pop(websocket, None)
except Exception as e:
logger.error(f"WebSocket error: {e}:{traceback.format_exc()}")
await self.broadcast_error({'message': f'{e}:{traceback.format_exc()}'})
# --- Run ---
def start(self) -> None:
async def start_async(self) -> None:
import uvicorn
uvicorn.run(self.app, host='0.0.0.0', port=self.port, log_level='info')
config = uvicorn.Config(self.app, host='0.0.0.0', port=self.port, log_level='info')
self._server = uvicorn.Server(config)
try:
await self._server.serve()
except asyncio.CancelledError:
self._server.should_exit = True
raise
def start(self) -> None:
asyncio.run(self.start_async())
def stop(self) -> None:
if self._server is None:
return
if self._event_loop is not None and self._event_loop.is_running():
self._event_loop.call_soon_threadsafe(lambda: setattr(self._server, "should_exit", True))
else:
self._server.should_exit = True
+20
View File
@@ -1,8 +1,28 @@
import subprocess
from pathlib import Path
from enironment import Environment
from steps.docker_plain import DockerImageBuild, DockerContainerCheck, DockerContainerDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
REPO_ROOT = Path(__file__).resolve().parents[2]
def _git_output(*args: str) -> str:
try:
return subprocess.check_output(
["git", "-C", str(REPO_ROOT), *args],
text=True,
stderr=subprocess.STDOUT,
).strip()
except subprocess.CalledProcessError as exc:
raise RuntimeError(f"Unable to read git metadata for nginx test config: {' '.join(args)}\n{exc.output}") from exc
CURRENT_BRANCH = _git_output("rev-parse", "--abbrev-ref", "HEAD")
CURRENT_COMMIT = _git_output("rev-parse", "HEAD")
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
container_check = DockerContainerCheck(
+52
View File
@@ -0,0 +1,52 @@
import asyncio
import sys
import types
from typing import Callable
import pytest
from enironment import Environment
import processing
from app import App
class TestAppAsyncio:
@pytest.mark.asyncio
async def test_run_web_async_stops_processing_when_server_exits(self, monkeypatch: pytest.MonkeyPatch) -> None:
process_calls = 0
emit_calls = 0
processing_started = asyncio.Event()
loop = asyncio.get_running_loop()
web_app_args: dict[str, object] = {}
def fake_process_all_jobs(environments: list[Environment], onupdate: Callable[[], None]) -> bool:
nonlocal process_calls
process_calls += 1
assert environments == []
onupdate()
loop.call_soon_threadsafe(processing_started.set)
return False
class FakeWebApp:
def __init__(self, core: App, port: int) -> None:
web_app_args["core"] = core
web_app_args["port"] = port
def emit_envs(self) -> None:
nonlocal emit_calls
emit_calls += 1
async def start_async(self) -> None:
await asyncio.wait_for(processing_started.wait(), timeout=1)
monkeypatch.setattr(processing, "process_all_jobs", fake_process_all_jobs)
monkeypatch.setitem(sys.modules, "web", types.SimpleNamespace(WebApp=FakeWebApp))
app = App({})
await asyncio.wait_for(app.runWebAsync(5001), timeout=1)
assert process_calls >= 1
assert emit_calls >= 1
assert web_app_args == {"core": app, "port": 5001}
assert app.emit_callback is None
+21 -21
View File
@@ -1,7 +1,7 @@
import asyncio
import json
import logging
import threading
from contextlib import suppress
from typing import Any, Generator
import docker
@@ -36,30 +36,30 @@ class TestDockerContainer:
logger.info(f"Starting")
app = App({"brencher_local1": brencher_local1.brencher_local1})
# app.runWeb(5001)
processing = threading.Thread(target=lambda: app.runWeb(5001), daemon=True)
processing.start()
processing = asyncio.create_task(app.runWebAsync(5001))
try:
eventually(
lambda: assert_equal(requests.get("http://localhost:5001/state", timeout=5000).status_code, 200, "Server did not respond with 200 OK within timeout"),
20.0,
1.0,
)
# await asyncio.sleep(5000)
state_data = requests.get("http://localhost:5001/state", timeout=5000).json()
logger.info(f"Application state: {json.dumps(state_data, indent=2)}")
eventually(
lambda: assert_equal(requests.get("http://localhost:5001/state", timeout=5000).status_code, 200, "Server did not respond with 200 OK within timeout"),
20.0,
1.0,
)
branches_data = requests.get("http://localhost:5001/branches", timeout=5000).json()
logger.info(f"Application branches: {json.dumps(branches_data, indent=2)}")
assert "brencher_local1" in state_data, "Missing 'brencher_local1' field in response"
assert len(state_data["brencher_local1"]["pipeline"]) > 0, "Steps defined for 'brencher_local1' are empty"
state_data = requests.get("http://localhost:5001/state", timeout=5000).json()
logger.info(f"Application state: {json.dumps(state_data, indent=2)}")
branches_data = requests.get("http://localhost:5001/branches", timeout=5000).json()
logger.info(f"Application branches: {json.dumps(branches_data, indent=2)}")
assert "brencher_local1" in state_data, "Missing 'brencher_local1' field in response"
assert len(state_data["brencher_local1"]["pipeline"]) > 0, "Steps defined for 'brencher_local1' are empty"
assert "brencher_local1" in branches_data, "Missing 'brencher_local1' field in response"
assert len(branches_data["brencher_local1"]) > 0, "No branches found for 'brencher_local1'"
assert "brencher_local1" in branches_data, "Missing 'brencher_local1' field in response"
assert len(branches_data["brencher_local1"]) > 0, "No branches found for 'brencher_local1'"
finally:
processing.cancel()
with suppress(asyncio.CancelledError):
await processing
assert processing.cancelled()
# TODO Verify DockerContainerDeploy step status is running and has correct image and ports
# {
+53 -41
View File
@@ -78,55 +78,67 @@ class TestDryRunPlaywright:
# Start app with nginx_local1 with main branch pre-selected and dry mode on.
env = nginx.test_local1
env.state.set_branches([("test/main", "HEAD")])
env.state.set_branches([(nginx.CURRENT_BRANCH, nginx.CURRENT_COMMIT)])
env.state.set_dry(True)
app = App({env.id: env})
_server_thread = threading.Thread(target=lambda: app.runWeb(APP_PORT), daemon=True)
_server_thread.start()
try:
# Wait for server to be up
eventually(lambda: assert_equal(requests.get(f"{APP_URL}/state", timeout=5).status_code, 200, "Server did not respond with 200 OK within timeout"))
logger.info("Server is up")
# Wait for server to be up
eventually(lambda: assert_equal(requests.get(f"{APP_URL}/state", timeout=5).status_code, 200, "Server did not respond with 200 OK within timeout"))
logger.info("Server is up")
# Wait for the pipeline to stabilize with dry=True:
# DockerContainerDeploy should return status="dry-run"
with brencher_page(APP_URL, env.id) as page:
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] == "SharedStateHolderInMemory"
)["status"]['dry'], True, "Pipeline step status is not 'running'")),
)
# Wait for the pipeline to stabilize with dry=True:
# DockerContainerDeploy should return status="dry-run"
with brencher_page(APP_URL, env.id) as page:
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] == "SharedStateHolderInMemory"
)["status"]['dry'], True, "Pipeline step status is not 'running'")),
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "dry-run")
logger.info("Pipeline is in dry-run mode")
# page.verify_pipeline_step_status("DockerContainerDeploy", "dry-run")
logger.info("Pipeline is in dry-run mode")
# Verify no real container was created
docker_client = docker.from_env()
real_containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(real_containers) == 0, "Container should NOT exist during dry run"
# Verify no real container was created
docker_client = docker.from_env()
real_containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(real_containers) == 0, "Container should NOT exist during dry run"
# Disable dry run from the UI.
page.verify_dry_run_on()
page.set_dry_run_off()
page.verify_dry_run_off()
# Disable dry run from the UI.
page.verify_dry_run_on()
page.set_dry_run_off()
page.verify_dry_run_off()
page.click_refresh() # TODO Remove it
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] =="SharedStateHolderInMemory"
)["status"]['dry'], False, "Pipeline step status is not 'running'")),
)
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] =="SharedStateHolderInMemory"
)["status"]['dry'], False, "Pipeline step status is not 'running'")),
)
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal({p["name"] for p in s[env.id]["pipeline"] if bool(p['is_running'])}, set(), "Pipeline still running")),
timeout=20,
interval=1.0,
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "running")
logger.info("Container deployed and running after dry run disabled")
# Wait for the real container to be created and started.
eventually(
lambda: _assert_container_running(docker_client),
# Allow time for a cold image build and first container start on CI runners.
timeout=180,
interval=5.0,
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "running")
logger.info("Container deployed and running after dry run disabled")
# Final sanity: real container must exist and be running
containers = docker_client.containers.list(filters={"name": CONTAINER_NAME})
assert len(containers) == 1, "Container should exist and be running after deploy"
assert containers[0].status == "running"
# Final sanity: real container must exist and be running
containers = docker_client.containers.list(filters={"name": CONTAINER_NAME})
assert len(containers) == 1, "Container should exist and be running after deploy"
assert containers[0].status == "running"
finally:
app.stop()
server_thread.join(timeout=30)
if server_thread.is_alive():
logger.warning("App server thread did not stop within 30 seconds")
def _assert_container_running(docker_client: docker.DockerClient) -> None:
containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(containers) > 0, "Container should exist after deploy"
assert_equal({container.status for container in containers}, {"running"}, "Container should be running after deploy")
+98
View File
@@ -101,6 +101,104 @@ class TestGitIntegration:
with pytest.raises(ValueError, match="requires either 'url' or 'remotes'"):
GitClone()
def test_git_clone_remotes_per_remote_prefix(self, repo_helper: RemoteRepoHelper) -> None:
# Per-remote `prefix` filters fetch refspec so only matching branches are visible.
# Two repos, two branches each (one matching the prefix, one not). Only the
# matching branch should appear in get_branches() for each remote.
def fresh_repo(prefix: str) -> str:
path = tempfile.mkdtemp(prefix=prefix)
repo = git.Repo.init(path, bare=False)
with repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
# Branches literally named `ansible/...` to match the prefix filter.
# create_commit creates an initial master, then branches off it.
repo_helper.create_commit(repo, "master", "master", "README.md", "content", f"{prefix} init")
repo_helper.create_commit(repo, "master", "ansible/master", "AN.md", "ansible", f"{prefix} ansible")
repo_helper.create_commit(repo, "master", "ansible/fix", "FIX.md", "fix", f"{prefix} fix")
return path
github_dir = fresh_repo("test_prefix_github_")
gitea_dir = fresh_repo("test_prefix_gitea_")
try:
clone = GitClone(remotes=[
{"name": "github", "url": github_dir, "prefix": "ansible"},
{"name": "gitea_admin", "url": gitea_dir, "prefix": "ansible"},
])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
branches = set(clone.get_branches())
# Per-remote refspec was applied: only ansible/* branches fetched.
assert "github/ansible/master" in branches, f"missing: {branches}"
assert "github/ansible/fix" in branches, f"missing: {branches}"
assert "gitea_admin/ansible/master" in branches, f"missing: {branches}"
assert "gitea_admin/ansible/fix" in branches, f"missing: {branches}"
# `master` exists upstream but is NOT visible (filtered by prefix).
assert "github/master" not in branches, f"unwanted: {branches}"
assert "gitea_admin/master" not in branches, f"unwanted: {branches}"
finally:
shutil.rmtree(github_dir, ignore_errors=True)
shutil.rmtree(gitea_dir, ignore_errors=True)
def test_git_clone_remotes_mixed_prefix(self, repo_helper: RemoteRepoHelper) -> None:
# Some remotes with prefix, others without. Each behaves independently:
# prefixed remotes only see matching branches, unprefixed see all.
def fresh_repo(prefix: str) -> str:
path = tempfile.mkdtemp(prefix=prefix)
repo = git.Repo.init(path, bare=False)
with repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
repo_helper.create_commit(repo, "master", "master", "README.md", "content", f"{prefix} init")
repo_helper.create_commit(repo, "master", "ansible/master", "AN.md", "ansible", f"{prefix} ansible")
repo_helper.create_commit(repo, "master", "feature_y", "OTHER.md", "other", f"{prefix} feature")
return path
github_dir = fresh_repo("test_mixed_github_")
gitea_dir = fresh_repo("test_mixed_gitea_")
try:
clone = GitClone(remotes=[
{"name": "github", "url": github_dir, "prefix": "ansible"},
{"name": "gitea_admin", "url": gitea_dir},
])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
branches = set(clone.get_branches())
# Prefixed: only ansible/master.
assert "github/ansible/master" in branches, f"missing: {branches}"
assert "github/feature_y" not in branches, f"unwanted: {branches}"
assert "github/master" not in branches, f"unwanted: {branches}"
# Unprefixed: all branches visible.
assert "gitea_admin/ansible/master" in branches, f"missing: {branches}"
assert "gitea_admin/feature_y" in branches, f"missing: {branches}"
assert "gitea_admin/master" in branches, f"missing: {branches}"
finally:
shutil.rmtree(github_dir, ignore_errors=True)
shutil.rmtree(gitea_dir, ignore_errors=True)
def test_git_clone_remotes_prefix_validation(self) -> None:
# `prefix` must be a string when set; non-string raises ValueError.
with pytest.raises(ValueError, match="'prefix' must be a string"):
GitClone(remotes=[
{"name": "github", "url": "git@example.com:a/b.git", "prefix": 123},
])
with pytest.raises(ValueError, match="'prefix' must be a string"):
GitClone(remotes=[
{"name": "github", "url": "git@example.com:a/b.git", "prefix": ["ansible"]},
])
# Empty string / None: accepted, treated as no prefix.
c1 = GitClone(remotes=[{"name": "github", "url": "git@example.com:a/b.git", "prefix": ""}])
assert c1.remotes[0]["prefix"] == ""
c2 = GitClone(remotes=[{"name": "github", "url": "git@example.com:a/b.git"}])
assert c2.remotes[0]["prefix"] == ""
def test_checkout_merged_one_branch(self, repo_helper: RemoteRepoHelper) -> None:
# Create main branch with initial commit
commit1 = repo_helper.create_commit(repo_helper.repo, "master", "master", "file1.txt", "content1",
+188
View File
@@ -0,0 +1,188 @@
"""
Regression tests for the WebSocket ws_connections race fixed in PR #42.
Background
----------
Before the fix, ``WebApp.ws_connections`` (a Dict[WebSocket, Dict[str, Any]])
was modified from two uncoordinated coroutines: ``websocket_endpoint`` and
``broadcast()``. Adding the key, then awaiting ``send_text``, then assigning
``self.ws_connections[websocket]["branches"] = ...`` left a window where the
key existed but the inner dict was empty. A concurrent ``broadcast()`` loop
could ``pop(ws)`` the same key, and the next assignment crashed with::
File "/app/backend/web.py", line 184, in websocket_endpoint
self.ws_connections[websocket]["branches"] = branches_payload
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: <starlette.websockets.WebSocket object at 0x...>
The fix serialises every mutation of ``ws_connections`` under an
``asyncio.Lock`` and registers the websocket with all three payloads in a
single atomic assignment before the first ``send_text`` await.
These tests verify the lock contract on a real ``WebApp`` instance with
fake websocket objects that simulate the production timing.
"""
import asyncio
import sys
from typing import Any, Dict, List
import pytest
# Match the project's import layout: backend modules are at top level.
sys.path.insert(0, "backend")
from web import WebApp
class FakeWebSocket:
"""Minimal stand-in for starlette.websockets.WebSocket.
- ``send_delay`` lets us widen the race window between accept and dict
writes, mimicking slow clients.
- ``receive_text`` raises ``WebSocketDisconnect`` so the endpoint exits
cleanly after one round-trip.
"""
def __init__(self, send_delay: float = 0.0) -> None:
self.sent: List[str] = []
self.accepted = False
self.send_delay = send_delay
async def accept(self) -> None:
self.accepted = True
async def send_text(self, message: str) -> None:
if self.send_delay:
await asyncio.sleep(self.send_delay)
self.sent.append(message)
async def receive_text(self) -> str:
from starlette.websockets import WebSocketDisconnect
raise WebSocketDisconnect()
class FakeCore:
"""Minimal stand-in for ``app.App`` — only the methods WebApp touches."""
def __init__(self) -> None:
self.local_envs: Dict[str, Any] = {}
def get_local_envs_to_emit(self) -> Dict[str, Any]:
return self.local_envs
def get_local_branches_to_emit(self) -> Dict[str, Any]:
return {}
def _make_web_app() -> WebApp:
return WebApp(core=FakeCore(), port=0) # type: ignore[arg-type]
@pytest.mark.asyncio
async def test_websocket_endpoint_registers_atomically_under_lock() -> None:
"""websocket_endpoint must register the ws with all three payloads in a
single dict assignment under ``ws_lock``, BEFORE any ``send_text``.
The lock and the atomic dict write are the contract; if either is
missing, this test fails with KeyError or with the dict missing keys.
"""
web_app = _make_web_app()
ws = FakeWebSocket(send_delay=0.01) # widen the race window
async def concurrent_broadcast() -> None:
# Try to interleave several broadcasts while the endpoint is
# awaiting send_text. Pre-fix, this would race with the dict
# updates and either crash or drop the connection.
for _ in range(5):
await asyncio.sleep(0.005)
await web_app.broadcast("branches", {"x": 1})
task = asyncio.create_task(concurrent_broadcast())
await web_app.websocket_endpoint(ws) # type: ignore[arg-type]
await task
# Endpoint must have accepted the connection
assert ws.accepted
# Three initial send_text calls (version, branches, environments)
assert len(ws.sent) >= 3
# After endpoint exits cleanly (WebSocketDisconnect), the entry is gone
assert ws not in web_app.ws_connections
@pytest.mark.asyncio
async def test_broadcast_and_concurrent_register_dont_crash() -> None:
"""``broadcast()`` and a concurrent ``websocket_endpoint`` registration
must serialise cleanly under the lock. Pre-fix this could raise
``RuntimeError: dictionary changed size during iteration`` or
``KeyError`` when the outer key was popped between accept and write.
"""
web_app = _make_web_app()
n = 20
sockets = [FakeWebSocket(send_delay=0.002) for _ in range(n)]
async def register(s: FakeWebSocket) -> None:
await web_app.websocket_endpoint(s) # type: ignore[arg-type]
async def broadcaster() -> None:
for _ in range(10):
await web_app.broadcast("environments", {"k": "v"})
await asyncio.sleep(0)
# All endpoints + broadcasts run concurrently. Pre-fix this would crash
# with KeyError or RuntimeError on most runs.
await asyncio.gather(
*(register(s) for s in sockets),
broadcaster(),
)
# All endpoints should have cleaned up after themselves.
async with web_app.ws_lock:
leftover = list(web_app.ws_connections.keys())
assert leftover == [], f"expected ws_connections to drain, got {leftover}"
@pytest.mark.asyncio
async def test_ws_lock_exists_and_is_asyncio_lock() -> None:
"""Sanity check on the lock attribute — guards against accidental
removal in future refactors.
"""
import asyncio as _asyncio
web_app = _make_web_app()
assert isinstance(web_app.ws_lock, _asyncio.Lock)
@pytest.mark.asyncio
async def test_broadcast_skips_already_sent_payload() -> None:
"""broadcast() must not re-send a payload it has already sent to a
connection (used for de-dup). This is the second contract of the
function (the first being serialised mutation); ensure the lock didn't
break the de-dup check.
"""
from web import custom_json_dumps
web_app = _make_web_app()
ws = FakeWebSocket()
# Pre-register a websocket with the EXACT JSON payload that broadcast()
# will produce. ``broadcast(event, data)`` JSON-encodes
# ``{event: data}`` into the message and compares against the stored
# value, so we need to mirror that here.
payload = {"branches": "msg-1"}
web_app.ws_connections[ws] = {"branches": custom_json_dumps(payload)}
# Two broadcasts with the same payload — second one must be de-duped
# against the stored value (the de-dup check on line 120:
# ``if self.ws_connections[websocket].get(event) == message: continue``).
await web_app.broadcast("branches", payload)
# After the first call, the de-dup is in place — a second identical
# broadcast must not reach send_text.
sent_before = len(ws.sent)
await web_app.broadcast("branches", payload)
assert len(ws.sent) == sent_before, (
f"de-dup broken: second broadcast sent {len(ws.sent) - sent_before} extra message(s)"
)
# Cleanup
async with web_app.ws_lock:
web_app.ws_connections.pop(ws, None)