Compare commits

...
Author SHA1 Message Date
hermes e0ae7de150 fix(web): serialise ws_connections mutations under asyncio.Lock
Python package / build (3.12.3) (pull_request) Failing after 1m3s
Python package / build (3.13) (pull_request) Failing after 56s
Python package / build (3.14) (pull_request) Failing after 1m0s
Race between websocket_endpoint and broadcast() in backend/web.py caused
intermittent 'KeyError: <WebSocket object>' errors every ~7s in
brencher-brencher-backend logs (counted 354+ in the last hour), forcing
the browser UI to reconnect and dropping log/sse updates.

Repro window:
  websocket_endpoint:
    175: self.ws_connections[websocket] = {}
    await websocket.send_text(version_payload)
    await websocket.send_text(branches_payload)
    184: self.ws_connections[websocket]["branches"] = branches_payload  # KeyError

  broadcast() runs concurrently, sends to N sockets, awaits each, and
  pops disconnected ones. The await yields control back to
  websocket_endpoint which then tries to write a key that was already
  popped (or — conversely — broadcast iterates over a snapshot while a
  parallel endpoint adds a new key it later tries to mutate).

Fix:
- Add asyncio.Lock to WebApp.__init__ and acquire it for every mutation
  of ws_connections (broadcast iteration + cleanup, websocket_endpoint
  registration, WebSocketDisconnect cleanup).
- websocket_endpoint now registers the websocket with all three
  payloads (version/branches/environments) in a single dict assignment
  BEFORE the first send_text await, so there is no window where the key
  exists but the inner dict is empty.
- broadcast() snapshots ws_connections.keys() via list() before iterating,
  and runs send_text + dict write + cleanup under the same lock.

Tests (tests/test_web_ws_lock.py, 4 cases):
- test_websocket_endpoint_registers_atomically_under_lock: a single
  endpoint connection with a slow send_text; concurrent broadcasts do
  not cause KeyError or a missing-key dict.
- test_broadcast_and_concurrent_register_dont_crash: 20 endpoints + a
  broadcaster all run concurrently; ws_connections must drain cleanly
  and never raise.
- test_ws_lock_exists_and_is_asyncio_lock: guard against future
  accidental removal.
- test_broadcast_skips_already_sent_payload: the de-dup check still
  works after the lock change (regression guard).

Refs #42
2026-09-05 17:52:44 +00:00
hermes 2556ac76f7 Merge remote-tracking branch 'admin/main' into HEAD
# Conflicts:
#	backend/web.py
#	tests/test_dry_run_playwright.py
2026-09-05 17:34:17 +00:00
hermesandadmin 99f8b73654 feat(git): per-remote branch prefix filtering for multi-remote GitClone (#31)
Merge main to all / rebase (push) Failing after 31s
Python package / build (3.12.3) (push) Failing after 2m28s
Python package / build (3.13) (push) Failing after 2m29s
Python package / build (3.14) (push) Failing after 2m38s
Python package / build (3.12.3) (pull_request) Failing after 2m17s
Python package / build (3.13) (pull_request) Canceled after 55s
Python package / build (3.14) (pull_request) Canceled after 0s
## Summary

`GitClone`'s top-level `branchNamePrefix` argument was silently ignored when constructed with the explicit `remotes=[...]` form (verified in `backend/steps/git.py:106`):

```python
if remote_name == "origin" and self.branchNamePrefix != "":
    # apply refspec
```

The prefix only fires when a remote is named `"origin"`. The `remotes=` form passes names through verbatim, so configs like the open PR #28 set the prefix but never apply it — all branches from all remotes are fetched.

This PR extends the API: each remote entry now optionally accepts a `prefix` field. When set, the fetch refspec for that remote becomes scoped to `<prefix>/*` branches. Backwards-compatible — remotes without `prefix` keep the existing behavior, and `url=` shorthand with `branchNamePrefix` still works.

## API

```python
clone = GitClone(remotes=[
    {"name": "github",       "url": "...", "prefix": "ansible"},
    {"name": "gitea_hermes", "url": "...", "prefix": "ansible"},
    {"name": "gitea_admin",  "url": "...", "prefix": "ansible"},
])
```

Each remote is independent — some can have `prefix`, others not.

## Changes

### `backend/steps/git.py`

- `__init__` (`remotes=` branch): validate `prefix` (string if set, reject non-string with `ValueError`), persist into the normalized entry. Empty string / None → no prefix applied (existing behavior).
- `progress()` fetch loop: replace the single `if remote_name == "origin"` check with per-entry logic. Effective prefix = `entry["prefix"] or (self.branchNamePrefix if remote_name == "origin" else "")`. Apply refspec per remote when effective prefix is non-empty.

### `tests/test_git_integration.py`

3 new tests in `TestGitIntegration`:

- `test_git_clone_remotes_per_remote_prefix` — both remotes prefixed; only `ansible/*` branches visible, `master` is filtered out.
- `test_git_clone_remotes_mixed_prefix` — one remote prefixed, one not; each behaves independently.
- `test_git_clone_remotes_prefix_validation` — `prefix=123` / `prefix=["ansible"]` raise `ValueError`; `prefix=""` and missing-`prefix` are accepted.

Existing `test_git_clone_named_remotes` continues to pass (verified — no behavior change for unprefixed remotes).

## Test results

```
29 passed in ~13s
```

(18 in `test_git_integration.py`, 11 in `test_dummy.py`/`test_caching_step.py`/`test_cached_env.py`.)

## Migration path

After merge:

- Open migration PRs (#26 gmail_mcp, #28 hermes, planned #19 gitea.py) — switch from `branchNamePrefix="ansible"` to `"prefix": "ansible"` on each remote.
- `hermes_infra.py` after PR #8 already drops `branchNamePrefix`, no change needed.
- Existing `url=` configs (immich, gmail_mcp-old, hermes-old, registry, gitea-old) — unaffected.

## Unblocks

PR #28 — once this lands, hermes.py can drop the (currently misleading) top-level `branchNamePrefix="ansible"` and add `"prefix": "ansible"` per remote.

Refs #30
Blocks #28
Part of #18 (parent migration ticket)

---------

Co-authored-by: admin <rudolfss13@gmail.com>
Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/31
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-05 19:20:44 +02:00
hermesandadmin 272504eab3 feat(configs): point gmail_mcp at gitea admin/hermes mcps repos (#26)
Merge main to all / rebase (push) Canceled after 0s
Python package / build (3.12.3) (push) Canceled after 0s
Python package / build (3.13) (push) Canceled after 0s
Python package / build (3.14) (push) Canceled after 0s
## Summary

Updates `backend/configs/gmail_mcp.py` to point at the new `mcps` stack mirrors on gitea instead of the legacy `uber_backup` repo.

Remotes now:

- `gitea_hermes` → `https://gitea.rudolf.keenetic.link/hermes/mcps`
- `gitea_admin`  → `https://gitea.rudolf.keenetic.link/admin/mcps`

Uses the `remotes=[{name, url}, …]` form introduced by #3.

## What changed vs the original #26 body

- **Repo switched**: `uber_backup` → `mcps` (the gmail + telegram MCP server repo, now mirrored on gitea under both `admin/` and `hermes/`).
- **GitHub remote dropped**: matches the pattern in `hermes_infra.py` after #8 and `docker_socket_proxy.py` after #23. The brencher container cannot reach `github.com` (squid returns HTTP 403 on CONNECT tunnel), so keeping a `github` remote would hard-fail the clone and trap brencher in the same retry loop that blocked #23 on `admin/reverseproxy` before its mirror was created.
- **`branchNamePrefix="immoscout"` dropped**: the prefix is only applied to remotes named `origin` (see `backend/steps/git.py` ~L104); neither remote qualifies, so the field is a silent no-op. Matches the cleanup done in `hermes_infra.py` / `brencher.py`.

## Diff (vs `main`)

```diff
-clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="immoscout")
+clone = GitClone(remotes=[
+    {"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/mcps"},
+    {"name": "gitea_admin",  "url": "https://gitea.rudolf.keenetic.link/admin/mcps"},
+])
```

## Prerequisites (already done)

Both target mirrors exist on gitea:

- `admin/mcps` — already present (primary, not a fork).
- `hermes/mcps` — forked in this session (id=10, fork of `admin/mcps`).

## Test plan

1. From the brencher state file for `gmail_mcp`, register `gitea_hermes/feat/migrate-gmail-mcp-to-multi-remote` as the desired branch.
2. Watch brencher logs: `Cloning repositories [...]` should succeed for both `gitea_hermes` and `gitea_admin`.
3. Confirm `gmail_mcp` stack redeploys against the new repo.

Refs #20
Part of #18

---------

Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/26
Co-authored-by: brencher_bot (hermes) <rudolfss13+hermes@gmail.com>
2026-09-05 19:20:16 +02:00
hermesandhermes fa9f8791bd feat(configs): point immich at gitea admin/hermes ansible repos (#12)
Merge main to all / rebase (push) Canceled after 0s
Python package / build (3.12.3) (push) Canceled after 0s
Python package / build (3.13) (push) Canceled after 0s
Python package / build (3.14) (push) Canceled after 0s
## Summary
Migrates `backend/configs/immich.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/immich.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git"[, branchNamePrefix=...])`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why migrate
- The old URL never resolved (`uber_backup.git` doesn't exist on gitea anyway) — so the deploy path was broken even though the file looked correct.
- `github.com` returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (commit `653aee9`).

## Scope decision: only `immich.py`
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) are being migrated in **separate PRs**, one per file. Each PR has its own branch + verification.

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/immich.py` — OK
- Real import via `from configs import immich`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''`
- pytest: **26/26** passed on the no-docker / no-playwright suite
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible`.
- end-to-end `docker stack deploy` — needs a real swarm.

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of this environment will clone from gitea instead of the unreachable github URL.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/12
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-05 19:19:47 +02:00
hermesandhermes 694c28109d feat(configs): point hermes at gitea admin/hermes ansible repos (#11)
Merge main to all / rebase (push) Canceled after 0s
Python package / build (3.12.3) (push) Canceled after 0s
Python package / build (3.13) (push) Canceled after 0s
Python package / build (3.14) (push) Canceled after 0s
## Summary
Migrates `backend/configs/hermes.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/hermes.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git"[, branchNamePrefix=...])`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why migrate
- The old URL never resolved (`uber_backup.git` doesn't exist on gitea anyway) — so the deploy path was broken even though the file looked correct.
- `github.com` returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (commit `653aee9`).

## Scope decision: only `hermes.py`
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) are being migrated in **separate PRs**, one per file. Each PR has its own branch + verification.

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/hermes.py` — OK
- Real import via `from configs import hermes`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''`
- pytest: **26/26** passed on the no-docker / no-playwright suite
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible`.
- end-to-end `docker stack deploy` — needs a real swarm.

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of this environment will clone from gitea instead of the unreachable github URL.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/11
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-05 19:19:22 +02:00
hermesandhermes 920249ec30 feat(configs): point gitea-config at gitea admin/hermes ansible repos (#10)
Merge main to all / rebase (push) Failing after 33s
Python package / build (3.12.3) (push) Successful in 2m30s
Python package / build (3.13) (push) Successful in 2m17s
Python package / build (3.14) (push) Successful in 2m23s
Python package / build (3.12.3) (pull_request) Successful in 2m40s
Python package / build (3.13) (pull_request) Successful in 2m40s
Python package / build (3.14) (pull_request) Successful in 2m50s
## Summary
Migrates `backend/configs/gitea.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/gitea.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git"[, branchNamePrefix=...])`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why migrate
- The old URL never resolved (`uber_backup.git` doesn't exist on gitea anyway) — so the deploy path was broken even though the file looked correct.
- `github.com` returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (commit `653aee9`).

## Scope decision: only `gitea.py`
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) are being migrated in **separate PRs**, one per file. Each PR has its own branch + verification.

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/gitea.py` — OK
- Real import via `from configs import gitea`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''`
- pytest: **26/26** passed on the no-docker / no-playwright suite
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible`.
- end-to-end `docker stack deploy` — needs a real swarm.

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of this environment will clone from gitea instead of the unreachable github URL.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/10
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-05 13:36:22 +02:00
hermes 75126cb2fd ci(merge_main): hardcode gitea.rudolf.keenetic.link (GITEA_SERVER_URL empty) (#47)
Merge main to all / rebase (push) Failing after 28s
Python package / build (3.12.3) (push) Successful in 2m33s
Python package / build (3.13) (push) Successful in 2m20s
Python package / build (3.14) (push) Successful in 2m16s
## Problem

Run #67 of `merge_main.yaml` failed with:

```
fatal: unable to access 'https:///admin/brencher.git/': URL rejected: No host part in the URL
```

The `PUSH_TOKEN` secret is now wired up correctly, but `${GITEA_SERVER_URL}` is empty in this Gitea Actions runtime (the variable the env-prefixed runner is supposed to set), so the substituted URL drops the host.

## Fix

Hardcode `gitea.rudolf.keenetic.link` in both `merge_main.yaml` and `merge_main_and_squash.yaml`. When the runner exposes `GITEA_SERVER_URL` in a future version, this can be replaced back with `${GITEA_SERVER_URL#https://}`.

## Supersedes

This is a follow-up to admin/brencher#46 which already wired `secrets.PUSH_TOKEN` but still had the empty env var.Reviewed-on: http://192.168.1.96/admin/brencher/pulls/47

Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-04 16:26:09 +02:00
admin 0e83458044 Обновить REST.instruction.md (#48)
Merge main to all / rebase (push) Failing after 12s
Python package / build (3.12.3) (push) Successful in 2m6s
Python package / build (3.13) (push) Successful in 2m10s
Python package / build (3.14) (push) Successful in 2m22s
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/48
2026-09-04 13:26:37 +02:00
admin 7dc36d7487 Merge pull request 'ci(merge_main): fix auth for self-hosted Gitea Actions' (#46) from hermes/brencher:fix/gitea-actions-merge-workflows into main
Merge main to all / rebase (push) Failing after 15s
Python package / build (3.12.3) (push) Successful in 2m17s
Python package / build (3.13) (push) Successful in 2m19s
Python package / build (3.14) (push) Successful in 2m23s
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/46
2026-09-04 13:08:06 +02:00
hermes b0e32f835a ci(merge_main): use PUSH_TOKEN instead of GITEA_PUSH_TOKEN
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (pull_request) Successful in 34s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (pull_request) Successful in 33s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (pull_request) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (pull_request) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (pull_request) Successful in 32s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (pull_request) Successful in 35s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (pull_request) Successful in 1m39s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (pull_request) Successful in 1m55s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (pull_request) Successful in 2m0s
Python package / build (pip list, 3.12.3) (pull_request) Successful in 31s
Python package / build (pip list, 3.13) (pull_request) Successful in 33s
Python package / build (pip list, 3.14) (pull_request) Successful in 32s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (pull_request) Successful in 32s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (pull_request) Successful in 29s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (pull_request) Successful in 29s
Match the secret name already created in admin/brencher.
2026-09-04 11:01:55 +00:00
hermes 1c0580750d ci(merge_main): fix auth for self-hosted Gitea Actions
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (pull_request) Successful in 2m13s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (pull_request) Successful in 35s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (pull_request) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (pull_request) Successful in 37s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (pull_request) Successful in 34s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (pull_request) Successful in 34s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (pull_request) Successful in 1m41s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (pull_request) Successful in 1m39s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (pull_request) Successful in 1m41s
Python package / build (pip list, 3.12.3) (pull_request) Successful in 31s
Python package / build (pip list, 3.13) (pull_request) Successful in 29s
Python package / build (pip list, 3.14) (pull_request) Successful in 32s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (pull_request) Successful in 27s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (pull_request) Successful in 29s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (pull_request) Successful in 29s
Both workflows were written for github.com and relied on the implicit
secrets.GITHUB_TOKEN + github.repository + a hardcoded github.com URL.
On this self-hosted Gitea instance (ghauth.rudolf.keenetic.link) they fail
with "Invalid username or token. Password authentication is not supported
for Git operations." against https://github.com/admin/brencher.git.

The fix:
- Read the push token from a new repo-level secret GITEA_PUSH_TOKEN
  (admin/brencher). Same value is injected into the checkout and into the
  rewritten remote URL so push can succeed.
- Build the origin URL from Gitea-injected GITHUB_REPOSITORY and
  GITEA_SERVER_URL instead of hardcoding github.com.
- Update git user.email/name for the Gitea Actions identity.

To enable: create secret GITEA_PUSH_TOKEN in admin/brencher (a PAT with
write access to the repo) via the Gitea UI before merging this PR.
2026-09-04 08:03:46 +00:00
8059cc74ea ci: speed up python-package.yml from ~13min to ~3-4min (#15)
Merge main to all / rebase (push) Failing after 14s
Python package / build (3.12.3) (push) Successful in 2m11s
Python package / build (3.13) (push) Successful in 2m22s
Python package / build (3.14) (push) Successful in 2m18s
## Summary
Optimizes `.github/workflows/python-package.yml` so the build job runs in ~3-4 minutes instead of ~13 minutes (which is what the recent run on PR #5 took — see https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/14, started 23:52, completed 00:05, **13m 19s wall-clock**).

## Why it was slow
The current matrix has **3 Python versions × 5 checkers = 15 separate CI jobs**, each one:
1. Checking out the repo (~10s)
2. Setting up Python via `actions/setup-python@v3` with **no cache enabled** (~15-30s)
3. Running `pip install -r requirements.txt -r requirements-dev.txt` **from scratch every single time** (~30-90s per job × 15 jobs = 7-22 minutes wasted)

The two debug entries (`python -c 'import sys; print(...)'` and `pip list`) were not real checks — they were noise from the upstream github `build-and-test-python` template, left in place but never reviewed.

## What's in this PR
- `.github/workflows/python-package.yml` — single file, +41/-56
  1. **`cache: 'pip'` enabled** with `cache-dependency-path: requirements.txt, requirements-dev.txt` — first run downloads and caches, subsequent runs hit the cache. Drops pip install from ~60s to ~5-15s when requirements don't change.
  2. **Two debug-only matrix entries removed** (`python -c ...` and `pip list`) — they were not real checks.
  3. **Matrix restructured**: one Python-version axis only, no checker axis. Each Python job runs `install + flake8 + mypy` sequentially. **15 jobs → 3 jobs** (~5x fewer fresh venvs to set up).
  4. **`concurrency: cancel-in-progress: true`** added — a new push to the same PR auto-cancels the previous in-flight run, saving runner minutes when iterating on a fix.
  5. Bumped `actions/setup-python` from `v3` to `v5` (required for `cache: 'pip'`).

## Verification
- YAML parses via PyYAML — ok
- `jobs` dict contains `['build']` only
- `build.strategy.matrix.python-version` = `['3.12.3', '3.13', '3.14']`
- **total CI jobs: 3** (was 15)
- `setup-python@v5` with `cache: 'pip'` + `cache-dependency-path` pointing at `requirements.txt` and `requirements-dev.txt`
- `concurrency` block correctly scoped to `ci-${{ github.workflow }}-${{ github.ref }}`

## Expected timing
| Scenario | Before | After |
|---|---|---|
| First run (cold cache) | ~13m 19s | ~5-7m |
| Subsequent runs (warm cache, no req changes) | ~13m 19s | ~3-4m |
| Re-run after `requirements.txt` change | ~13m 19s | ~4-5m |

## What was NOT verified
- Actual Gitea Actions runner behaviour with `cache: 'pip'` — the Gitea Actions runner is GitHub Actions-compatible, and `actions/setup-python@v5` with `cache: 'pip'` is a well-known idiom there, but the local Gitea runner config may or may not honour it. If it doesn't, the 15→3 fan-out reduction alone is still a ~5x speedup.

## Note on PR #14
PR #14 ("re-enable the tests job") is still open. If/when it merges, this PR will need a follow-up to re-add the `tests:` job with the same `cache: 'pip'` setup applied — otherwise the playwright install will dominate again.

## Commit history
```
* db3807e (HEAD -> feat/optimize-ci-build-time, origin/feat/optimize-ci-build-time)
      ci: speed up python-package.yml from ~13min to ~3-4min
* 70f4a41 ci: comment out the tests job in python-package.yml (#13)   (upstream/main tip)
```

---------

Co-authored-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/15
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-04 09:48:29 +02:00
hermesandadmin e690f6a50c feat(configs): point torrserv_proxy at gitea admin/hermes repos (#23)
Merge main to all / rebase (push) Failing after 18s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 56s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 48s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 46s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 44s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 42s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 42s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m56s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 2m0s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m2s
Python package / build (pip list, 3.12.3) (push) Successful in 39s
Python package / build (pip list, 3.13) (push) Successful in 39s
Python package / build (pip list, 3.14) (push) Successful in 55s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 48s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 33s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 43s
## Summary

Migrates `backend/configs/torrserv_proxy.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/reverseproxy.git`) to the new `remotes=[{name, url}, ...]` form introduced by #3.

Source repo is `rudolf1/reverseproxy` (not `brencher`), so the three remotes point at:

- `github`       → `https://github.com/rudolf1/reverseproxy.git`
- `gitea_admin`  → `https://gitea.rudolf.keenetic.link/admin/reverseproxy`
- `gitea_hermes` → `https://gitea.rudolf.keenetic.link/hermes/reverseproxy`

Same pattern that already works in production for `brencher.py` and `brencher2.py`.

## Diff

```diff
-clone = GitClone(url="https://github.com/rudolf1/reverseproxy.git")
+clone = GitClone(remotes=[
+	{"name": "github",       "url": "https://github.com/rudolf1/reverseproxy.git"},
+	{"name": "gitea_admin",  "url": "https://gitea.rudolf.keenetic.link/admin/reverseproxy"},
+	{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/reverseproxy"},
+])
```

## Test plan

Manual test on prod: trigger brencher with this config, verify it clones from the gitea mirrors and `torrserv_proxy` stack deploys successfully.

Refs #22
Part of #18

---------

Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/23
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-30 22:29:00 +02:00
hermes 98ee950a1f fix(docker): cap swarm config name at 64 chars via sha256 fallback (#17)
Merge main to all / rebase (push) Failing after 14s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 39s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 46s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 39s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 39s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 38s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 41s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m50s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m57s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m5s
Python package / build (pip list, 3.12.3) (push) Successful in 34s
Python package / build (pip list, 3.13) (push) Successful in 34s
Python package / build (pip list, 3.14) (push) Successful in 35s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 31s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 31s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 32s
## What

Since #5 (Fix/show pipeline errors), brencher appends a version tail to every swarm config name so each deploy creates a new config object (swarm configs are immutable). For long base names plus the version string, the resulting prefixed name (stack_name_ + name) can exceed docker swarm's 64-char limit and `docker stack deploy` fails with `InvalidArgument`.

## Repro

```
[2026-08-29 19:38:46,063] failed to create config hermes_infra_brencher_state_proxy_conf_auto-16875f17-4cb8eeae-bfb3c220: Error response from daemon: rpc error: code = InvalidArgument desc = invalid name, only 64 [a-zA-Z0-9-_.] characters allowed
```

`brencher_state_proxy_conf_auto-16875f17-4cb8eeae-bfb3c220` is 57 chars, plus the 13-char `hermes_infra_` stack prefix = 70, over the 64-char limit.

## Fix

When the would-be full prefixed name is longer than 64 chars, replace the suffix with `sha256(full)[:32]`. 32 hex chars is comfortably unique per `(base_name, version)` pair and the prefixed name ends up at 12 + 1 + 32 = 45 chars, well under the limit.

Short names keep the readable `<name>_<version>` form (no change for the common case).

## Verified

Manual sanity check on the four configs currently in `hermes_squid`:

| base | version | compose name | docker name | length |
|---|---|---|---|---|
| squid_conf | auto-4cb8eeae-bfb3c220 | squid_conf_auto-4cb8eeae-bfb3c220 | hermes_infra_squid_conf_auto-4cb8eeae-bfb3c220 | 46 |
| brencher_state_proxy_conf | auto-16875f17-4cb8eeae-bfb3c220 | 7644a2983c7f6c7b09c6ed84bff0096d | hermes_infra_7644a2983c7f6c7b09c6ed84bff0096d | 45 |
| brencher_state_proxy_conf | auto-4cb8eeae-bfb3c220 | brencher_state_proxy_conf_auto-4cb8eeae-bfb3c220 | hermes_infra_brencher_state_proxy_conf_auto-4cb8eeae-bfb3c220 | 61 |
| nginx_hermes_conf | auto-16875f17-4cb8eeae-bfb3c220 | nginx_hermes_conf_auto-16875f17-4cb8eeae-bfb3c220 | hermes_infra_nginx_hermes_conf_auto-16875f17-4cb8eeae-bfb3c220 | 62 |

All under 64 chars; sha256 fallback only kicks in for the long case.

## Impact

- Fixes deploys of stacks that have a config with a long base name (e.g. `brencher_state_proxy_conf` in `hermes_infra`).
- Other stacks (`gitea`, `immich`, `torrserv_proxy`, etc.) are unaffected because their config names are short enough to fit the existing readable form.
- After merge + brencher self-deploy, the failed hermes_infra stack at `brencher-state-proxy` (failed since 2026-08-26) can finally be re-deployed and pick up the new `brencher-state-proxy.conf` with the resolver fix from #5's companion config rename.

## Related

- This unblocks `admin/ansible#21` (feat/stt-voice-faster-whisper) whose deploy was failing on the same length error.
- The companion nginx-side fix in `admin/ansible#21` (`brencher-state-proxy.conf`: `resolver` + `set $upstream`) will start working once this lands and the config object is actually recreated.Reviewed-on: http://192.168.1.96/admin/brencher/pulls/17
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-29 22:54:07 +02:00
admin e4f90b1f23 Fix/show pipeline errors (#5)
Merge main to all / rebase (push) Failing after 22s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 37s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 31s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 42s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 43s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m53s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m58s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 1m54s
Python package / build (pip list, 3.12.3) (push) Successful in 37s
Python package / build (pip list, 3.13) (push) Successful in 37s
Python package / build (pip list, 3.14) (push) Successful in 34s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 29s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 36s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 35s
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/5
2026-08-27 10:58:55 +02:00
hermesandhermes ba17865a49 feat(configs): point hermes_infra at gitea admin/hermes ansible repos (#8)
Merge main to all / rebase (push) Failing after 15s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 41s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 36s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 41s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 35s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 37s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m43s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m55s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 1m48s
Python package / build (pip list, 3.12.3) (push) Successful in 34s
Python package / build (pip list, 3.13) (push) Successful in 32s
Python package / build (pip list, 3.14) (push) Successful in 36s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 37s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 33s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 31s
## Summary
Switches `backend/configs/hermes_infra.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/hermes_infra.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why this config specifically
- `hermes_infra.py` is the config that **deploys `hermes_squid/stack-compose.yml`** — i.e. it is what brings the sidecars added in `admin/ansible#1` (the docker_socket_proxy + brencher-state-proxy from earlier today) to a running swarm. Pointing its clone at the unreachable github URL meant the deploy path was broken even though the file looked correct.
- `uber_backup.git` doesn't exist on gitea anyway (only `admin/ansible`, `hermes/ansible`, `admin/brencher`, `hermes/brencher` exist) — so the old URL was always wrong, not just unreachable.
- github.com returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (PR #4 / commit `653aee9`).

## Scope decision: only `hermes_infra.py` was migrated
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) still use the legacy form. They're intentionally left untouched — this PR keeps the migration scope narrow on `hermes_infra` only, as requested. A future follow-up PR can migrate them once the upstream uber_backup / reverseproxy mirrors exist on gitea (they don't right now).

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/hermes_infra.py` — OK
- Real import via `from configs import hermes_infra`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''` (not just unset)
  - `env.id == "hermes_infra"`, pipeline unchanged
  - `deploy step docker_compose_path == "hermes_squid/stack-compose.yml"`
  - `deploy step stack_name == "hermes_infra"`
- pytest: **26/26** passed on the no-docker / no-playwright suite (`test_dummy`, `test_caching_step`, `test_cached_env`, `test_git_integration`)
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`
  - no `github.com` in any remote URL

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible` (the live-deploy source). Sandbox can't simulate this.
- end-to-end `docker stack deploy hermes_infra` — needs a real swarm

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of the `hermes_infra` environment will clone from gitea instead of the unreachable github URL. Combined with `admin/ansible#1` (which adds the new services to `hermes_squid/stack-compose.yml`) and `admin/brencher#4` (which registers `docker_socket_proxy` in the PROFILES env var), the full chain is:
1. brencher clones ansible from `gitea.rudolf.keenetic.link/{hermes,admin}/ansible`
2. Checks out a merged branch containing the new sidecars
3. Builds (only `sarg` — `docker_socket_proxy` and `brencher-state-proxy` use upstream images, no build)
4. `docker stack deploy hermes_infra` rolls out the full stack

## Commit history
```
* 9a0cd90 (HEAD -> feat/point-hermes-infra-at-gitea, origin/feat/point-hermes-infra-at-gitea)
      feat(configs): point hermes_infra at gitea admin/hermes ansible repos
* e533174 Allow naming each GitClone remote explicitly (#3)         (upstream/main tip)
```

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/8
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-26 11:30:17 +02:00
hermesandhermes 70f4a4163e ci: comment out the tests job in python-package.yml (#13)
## Summary
Comments out the `tests:` job in `.github/workflows/python-package.yml`. The slow `pytest` matrix (with `playwright install --with-deps chromium`) no longer runs on every push or PR — only the cheap `build` job (lint + mypy) remains.

## Why
- `pytest` requires `playwright install --with-deps chromium` on every run, which dominates the CI time even though pytest is not currently the merge gate.
- The `build` job (flake8 + mypy) still runs and still catches real regressions cheaply.
- `workflow_dispatch` is unchanged — tests can still be triggered manually from the Actions UI when needed.

## What's in this PR
- `.github/workflows/python-package.yml` — single file, +35/-34
  - The `tests:` job (33 lines) is now commented with leading `# `
  - Everything else (triggers, `build:` job, matrix) is unchanged
  - YAML still parses cleanly; `jobs` dict now contains only `build`

## Verification
- `python -c "import yaml; yaml.safe_load(open('.github/workflows/python-package.yml'))"` — OK
- `wf['jobs']` contains `['build']` only (no `tests` key)
- `build.matrix.checker` still has all 5 checkers:
  - `python -c "import sys; ..."` (debug)
  - `pip list` (debug)
  - `flake8 . --select=E9,F63,F7,F82` (syntax errors)
  - `flake8 . --exit-zero --max-complexity=10` (complexity)
  - `mypy --no-incremental --config-file mypy.ini backend tests`
- triggers unchanged: `workflow_dispatch`, `push`, `pull_request` (all targeting `main`)

## How to re-enable
Remove the leading `# ` from each commented line, or delete the commented block entirely and rely on git history.

## What was NOT verified
- The actual GitHub Actions runner behaviour change — this repo runs on Gitea Actions (the project's primary CI is the brencher pipeline itself, not Gitea-hosted runners for these workflows). Gitea's Actions runner is GitHub Actions–compatible and will see the `tests` job as absent and skip it.

## Commit history
```
* e271b2e (HEAD -> feat/disable-ci-tests, origin/feat/disable-ci-tests)
      ci: comment out the tests job in python-package.yml
* e533174 Allow naming each GitClone remote explicitly (#3)         (upstream/main tip)
```

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/13
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-25 23:24:49 +02:00
e5331745be Allow naming each GitClone remote explicitly (#3)
Merge main to all / rebase (push) Failing after 19s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 38s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 38s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 52s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 41s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 40s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m55s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 2m1s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m1s
Python package / build (pip list, 3.12.3) (push) Successful in 38s
Python package / build (pip list, 3.13) (push) Successful in 40s
Python package / build (pip list, 3.14) (push) Successful in 41s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 35s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 37s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 40s
Python package / tests (pytest, 3.12.3) (push) Failing after 7m43s
Python package / tests (pytest, 3.13) (push) Failing after 7m7s
Python package / tests (pytest, 3.14) (push) Failing after 7m36s
Implements the review comment on #2:

> I would like to give names for remotes

## Summary

Adds a `remotes=[{"name": ..., "url": ...}, ...]` constructor to `GitClone` so the config author controls how each URL maps to a git remote. The existing `url=str` / `url=[str]` shorthand is kept as backward-compatible (positional list still maps to `origin`, `repo1`, `repo2`, ...), so PR #2's `test_git_clone_multiple_urls` continues to pass.

## Changes

- **`backend/steps/git.py`** — `GitClone.__init__` now accepts either `url=` or `remotes=`. With `remotes=`, every entry must carry a non-empty unique `name` and `url`; duplicates and missing fields raise `ValueError`. The constructor rejects passing both or neither. `self.remotes` becomes the source of truth; `self.urls` is a derived list (kept for any introspection). `CheckoutMerged.find_desired_commits` falls back to the configured primary remote instead of the hardcoded `'origin'` when a branch key has no remote prefix, so configs that don't name a remote `origin` still work.
- **`backend/configs/brencher.py`, `brencher2.py`** — switch from positional list to named remotes `github`, `gitea_admin`, `gitea_hermes`.
- **`tests/test_git_integration.py`** — adds `test_git_clone_named_remotes` (asserts the user-chosen names appear verbatim and `origin`/`repo1` do not leak) and `test_git_clone_remotes_validation` (covers every rejection path: empty list, missing/empty name, missing/empty url, duplicate name, both-kwargs, no-kwargs).

## Diff vs #2

```
backend/configs/brencher.py    |  8 ++---
backend/configs/brencher2.py   |  8 ++---
backend/steps/git.py           | 76 +++++++++++++++++++++++++++++++++++++++-------
tests/test_git_integration.py  | 75 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 147 insertions(+), 20 deletions(-)
```

Targets `admin/multiple_remotes` so this lands on top of #2. After #2 merges, this is a clean follow-up; if #2 is reworked, this branch will need a rebase.

## Verification

- `python3 -m py_compile` on all 4 changed files: clean.
- AST-driven call-shape check: every `GitClone(...)` in both configs uses the new `remotes=` kwarg with exactly `{github, gitea_admin, gitea_hermes}`; no positional-list leftover in configs.
- Executed the new `__init__` in isolation against a stubbed `super()` and exercised 12 cases: 3 OK shapes (url=str, url=[..], remotes=[valid]) and 9 rejection shapes (empty list, missing/empty name, missing/empty url, duplicate name, both-kwargs, no-kwargs, non-dict entry). All 12 produced the asserted `ValueError`/`TypeError` with the expected message substring.
- Could not run `pytest` in this sandbox (PyPI is blocked by the proxy and no `pytest`/`gitpython` is pre-installed). The two new tests and PR #2's existing test should be run in a real env before merge.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Co-authored-by: rudolf <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/3
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-25 18:53:12 +02:00
admin 94a858d8ef Version on UI + gitea 2026-08-12 21:09:07 +02:00
admin 57393c933c hermes (#83)
* hermes
2026-08-05 21:26:07 +02:00
copilot-swe-agent[bot]andrudolf1 0e43c0fd13 harden playwright test config and assertions
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (pull_request) Successful in 36s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (pull_request) Successful in 34s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (pull_request) Successful in 36s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (pull_request) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (pull_request) Successful in 35s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (pull_request) Successful in 36s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (pull_request) Successful in 1m43s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (pull_request) Successful in 1m54s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (pull_request) Successful in 1m59s
Python package / build (pip list, 3.12.3) (pull_request) Successful in 31s
Python package / build (pip list, 3.13) (pull_request) Successful in 33s
Python package / build (pip list, 3.14) (pull_request) Successful in 31s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (pull_request) Successful in 32s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (pull_request) Successful in 36s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (pull_request) Successful in 31s
Python package / tests (pytest, 3.12.3) (pull_request) Canceled after 0s
Python package / tests (pytest, 3.13) (pull_request) Canceled after 0s
Python package / tests (pytest, 3.14) (pull_request) Canceled after 0s
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/d019d010-3ca0-4a73-8826-620eef58edab

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 20:08:12 +00:00
copilot-swe-agent[bot]andrudolf1 ab01fca841 stabilize dry-run playwright integration
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/d019d010-3ca0-4a73-8826-620eef58edab

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 20:06:11 +00:00
copilot-swe-agent[bot]andrudolf1 b2782ee220 clarify async taskgroup behavior
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:36:56 +00:00
copilot-swe-agent[bot]andrudolf1 1acea85934 add async shutdown cleanup and review fixes
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:35:59 +00:00
copilot-swe-agent[bot]andrudolf1 6f3dd6ebd8 refactor app concurrency to asyncio taskgroup
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:34:46 +00:00
copilot-swe-agent[bot] efe49119a3 Initial plan 2026-05-19 19:02:30 +00:00
29 changed files with 1228 additions and 275 deletions
+4 -8
View File
@@ -20,16 +20,16 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
token: ${{ secrets.PUSH_TOKEN }}
- name: Configure git
run: |
git config user.email "github-actions[bot]@users.noreply.github.com"
git config user.name "GitHub Actions"
git config user.email "gitea-actions[bot]@noreply.localhost"
git config user.name "Gitea Actions"
- name: Fetch all branches
run: |
git remote set-url origin "https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
git remote set-url origin "https://x-access-token:${PUSH_TOKEN}@gitea.rudolf.keenetic.link/${GITHUB_REPOSITORY}.git"
git fetch --all --prune
- name: Merge every branch onto main
@@ -37,7 +37,6 @@ jobs:
run: |
set -eo pipefail
MAIN=main
# Collect remote branch names (exclude HEAD and main)
branches=$(git for-each-ref --format='%(refname:short)' refs/remotes/origin/ \
| sed 's|^origin/||' \
| grep -v -e '^HEAD$' -e '^main$' -e '^origin$' \
@@ -49,11 +48,8 @@ jobs:
continue
fi
echo "=== Processing branch: $br ==="
# Create/update a local branch from remote
git checkout -B "$br" "origin/$br"
# Merge origin/main into branch (non-interactive)
if git merge --no-edit "origin/$MAIN"; then
# Push updated branch to remote (no force)
if git push origin "$br"; then
echo "Pushed merged $br"
else
+4 -8
View File
@@ -18,16 +18,16 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
token: ${{ secrets.PUSH_TOKEN }}
- name: Configure git
run: |
git config user.email "github-actions[bot]@users.noreply.github.com"
git config user.name "GitHub Actions"
git config user.email "gitea-actions[bot]@noreply.localhost"
git config user.name "Gitea Actions"
- name: Fetch all branches
run: |
git remote set-url origin "https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
git remote set-url origin "https://x-access-token:${PUSH_TOKEN}@gitea.rudolf.keenetic.link/${GITHUB_REPOSITORY}.git"
git fetch --all --prune
- name: Merge every branch onto main
@@ -35,7 +35,6 @@ jobs:
run: |
set -eo pipefail
MAIN=main
# Collect remote branch names (exclude HEAD and main)
branches=$(git for-each-ref --format='%(refname:short)' refs/remotes/origin/ \
| sed 's|^origin/||' \
| grep -v -e '^HEAD$' -e '^main$' -e '^origin$' \
@@ -47,12 +46,9 @@ jobs:
continue
fi
echo "=== Processing branch: $br ==="
# Create/update a local branch from remote
git reset --hard HEAD
git checkout -B "$br" "origin/$br"
# Merge origin/main into branch (non-interactive)
if git merge --no-edit --squash "origin/$MAIN"; then
# Push updated branch to remote (no force)
git reset --soft "origin/$MAIN"
git commit -m "Squash"
if git push origin "$br" --force-with-lease; then
+18 -54
View File
@@ -1,6 +1,3 @@
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
name: Python package
on:
@@ -10,73 +7,40 @@ on:
pull_request:
branches: [ "main" ]
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: [
# "3.11.3",
"3.12.3",
"3.13",
"3.14",
# "3.15",
]
checker: [
"python -c \"import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')\"",
"pip list",
"flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics",
"flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics",
# "ty check backend",
"mypy --no-incremental --config-file mypy.ini backend tests",
]
python-version: ["3.12.3", "3.13", "3.14"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}, ${{ matrix.checker }}
uses: actions/setup-python@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: |
requirements.txt
requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -r requirements-dev.txt
- name: Run checker
run: ${{ matrix.checker }}
tests:
runs-on: ubuntu-latest
needs: build
strategy:
fail-fast: false
matrix:
python-version: [
# "3.11.3",
"3.12.3",
"3.13",
"3.14",
# "3.15",
]
checker: [
"pytest",
]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}, ${{ matrix.checker }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
- name: Lint (flake8)
run: |
# sudo apt-get update
# sudo apt-get upgrade -y
# sudo apt-get install -y --no-install-recommends git docker.io
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
playwright install --with-deps chromium
- name: Run checker
run: "pytest"
flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Type-check (mypy)
run: mypy --no-incremental --config-file mypy.ini backend tests
+3 -9
View File
@@ -1,14 +1,10 @@
FROM nginx:1.27-alpine
COPY .git /tmp/repo/.git
# Configure nginx to listen on 5001 to match container port expectations.
RUN sed -i 's/listen 80;/listen 5001;/' /etc/nginx/conf.d/default.conf
# Build a custom landing page directly in the image and inject git revision.
# Build a custom landing page directly in the image.
RUN set -eux; \
apk add --no-cache --virtual .git-deps git; \
GIT_REVISION="$(git --git-dir=/tmp/repo/.git rev-parse --short HEAD 2>/dev/null || echo unknown)"; \
{ \
echo '<!doctype html>'; \
echo '<html lang="en">'; \
@@ -21,11 +17,9 @@ RUN set -eux; \
echo ' <h1>Nginx Is Running</h1>'; \
echo ' <p>Custom landing page for the Brencher test app container.</p>'; \
echo ' <p>Port: 5001</p>'; \
echo " <p>Git revision: ${GIT_REVISION}</p>"; \
echo ' <p>Git revision: unknown</p>'; \
echo '</body>'; \
echo '</html>'; \
} > /usr/share/nginx/html/index.html; \
rm -rf /tmp/repo/.git; \
apk del .git-deps
} > /usr/share/nginx/html/index.html
EXPOSE 5001
+3
View File
@@ -17,3 +17,6 @@
state: str # 'Active' or 'Pause'
pipeline: Dict[str, Any]
+74 -23
View File
@@ -1,8 +1,8 @@
import asyncio
import logging
import os
import signal
import sys
import threading
import traceback
from typing import Any, Callable, Dict, List, Optional
@@ -27,9 +27,12 @@ class App:
def __init__(self, environments: Dict[str, Environment]) -> None:
self.environments: Dict[str, Environment] = {id: wrap_in_cached(e) for id, e in environments.items()}
self.state_lock = threading.Lock()
self.environment_update_event = threading.Event()
self.environment_update_event = asyncio.Event()
self.environment_update_version = 0
self.emit_callback: Optional[Callable[[], None]] = None
self.reset_requested = False
self.shutdown_event = asyncio.Event()
self.web_app: Any = None
def get_local_envs_to_emit(self) -> Dict[str, Dict[str, Any]]:
env_dtos: Dict[str, Dict[str, Any]] = {}
@@ -48,7 +51,7 @@ class App:
"name": r.name,
"status": [str(result), stack],
"error": True,
"is_running": True,
"is_running": False,
})
else:
pipeline_state.append({
@@ -62,7 +65,7 @@ class App:
"name": r.name,
"status": [str(e), stack],
"error": True,
"is_running": True,
"is_running": False,
})
env_dtos[env.id] = {'id': env.id }
env_dtos[env.id]['pipeline'] = pipeline_state
@@ -88,35 +91,83 @@ class App:
return branches
def processing_thread(self) -> None:
while True:
async def processing_loop(self) -> None:
while not self.shutdown_event.is_set():
import processing
emit = self.emit_callback or (lambda: None)
update_version = self.environment_update_version
with self.state_lock:
logger.info(f"Processing")
if processing.process_all_jobs(list(self.environments.values()), emit):
self.environment_update_event.wait(timeout=1 * 5)
else:
self.environment_update_event.wait(timeout=1 * 60)
self.environment_update_event.clear()
logger.info("Processing")
if self.reset_requested:
await asyncio.to_thread(processing.reset_caches, list(self.environments.values()))
self.reset_requested = False
has_error = await asyncio.to_thread(processing.process_all_jobs, list(self.environments.values()), emit)
if self.shutdown_event.is_set():
return
# Skip the wait when a newer update arrived while the current processing pass was running.
if self.environment_update_version != update_version:
continue
timeout = 5 if has_error else 60
try:
await asyncio.wait_for(self.environment_update_event.wait(), timeout=timeout)
except asyncio.TimeoutError:
pass
self.environment_update_event.clear()
def run(self) -> None:
processing = threading.Thread(target=self.processing_thread)
processing.daemon = True
processing.start()
def notify_environment_update(self) -> None:
self.environment_update_version += 1
self.environment_update_event.set()
def request_reset(self) -> None:
self.reset_requested = True
def stop(self) -> None:
self.shutdown_event.set()
self.notify_environment_update()
if self.web_app is not None:
self.web_app.stop()
async def runAsync(self) -> None:
await self.processing_loop()
async def runHeadlessAsync(self) -> None:
await self.processing_loop()
def runHeadless(self) -> None:
"""Run the processing loop on the current thread; blocks forever."""
self.processing_thread()
"""Run the processing loop until cancelled; blocks forever."""
asyncio.run(self.runHeadlessAsync())
def runWeb(self, port: int) -> None:
async def _run_web_server(self, web_app: Any) -> None:
"""Raise a sentinel exception when the server stops so TaskGroup cancels processing."""
await web_app.start_async()
raise _WebServerStopped()
async def runWebAsync(self, port: int) -> None:
import web
web_app = web.WebApp(core=self, port=port)
self.web_app = web_app
self.emit_callback = web_app.emit_envs
self.run()
web_app.start()
try:
# Keep processing and web serving coupled so shutdown or fatal failure tears down the whole app.
async with asyncio.TaskGroup() as task_group:
task_group.create_task(self.processing_loop())
task_group.create_task(self._run_web_server(web_app))
except* _WebServerStopped:
pass
finally:
self.emit_callback = None
self.web_app = None
def runWeb(self, port: int) -> None:
asyncio.run(self.runWebAsync(port))
def run(self) -> None:
asyncio.run(self.runAsync())
class _WebServerStopped(Exception):
pass
if __name__ == '__main__':
cli_env_ids_list = sys.argv[1:]
+10 -6
View File
@@ -5,7 +5,11 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/brencher.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/brencher"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/brencher"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher",
@@ -45,10 +49,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="docker-compose.yml",
)
checkPing = UrlCheck(
url="https://brencher.rudolf.keenetic.link/state",
expected=checkPingF,
)
#checkPing = UrlCheck(
# url="https://brencher.rudolf.keenetic.link/state",
# expected=checkPingF,
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://brencher.rudolf.keenetic.link/",
@@ -68,7 +72,7 @@ brencher = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
]
)
+10 -6
View File
@@ -21,7 +21,11 @@ def checkPingF(obj: Any) -> None:
raise Exception(f"brencher2 check failed for: {v}")
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/brencher.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/brencher"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/brencher"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher2",
@@ -72,10 +76,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="docker-compose.yml",
)
checkPing = UrlCheck(
url="https://brencher.rudolf.keenetic.link/state",
expected=checkPingF,
)
#checkPing = UrlCheck(
# url="https://brencher.rudolf.keenetic.link/state",
# expected=checkPingF,
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://brencher.rudolf.keenetic.link/",
@@ -98,7 +102,7 @@ brencher2 = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
],
)
+80
View File
@@ -0,0 +1,80 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="gitea",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=None,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="gitea",
docker_compose_path="poc/gitea/stack-compose.yml",
)
# checkPing1 = UrlCheck(
# url="http://100.70.193.97:8087/api/status",
# expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
# )
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://gitea.rudolf.keenetic.link",
}
})
__all__ = ["gitea"]
gitea = Environment(
id="gitea",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
# checkPing1,
logUrls
]
)
+6 -3
View File
@@ -4,7 +4,10 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="immoscout")
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/mcps"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/mcps"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="gmail_mcp",
@@ -23,7 +26,7 @@ checkoutMerged = CheckoutMerged(clone,
buildDocker = DockerComposeBuild(clone,
docker_repo_username="",
docker_repo_password="",
docker_compose_path="gmail-mcp-server/docker-compose.yml",
docker_compose_path="docker-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
envs=lambda: {
@@ -39,7 +42,7 @@ deployDocker = DockerSwarmDeploy(
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="gmail_mcp",
docker_compose_path="gmail-mcp-server/docker-compose.yml",
docker_compose_path="docker-compose.yml",
)
checkPing1 = UrlCheck(
+80
View File
@@ -0,0 +1,80 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="hermes",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=None,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="hermes",
docker_compose_path="poc/hermes/stack-compose.yml",
)
checkPing1 = UrlCheck(
url="http://100.70.193.97:8087/api/status",
expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://hermes.rudolf.keenetic.link",
}
})
__all__ = ["hermes"]
hermes = Environment(
id="hermes",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing1,
logUrls
]
)
+94
View File
@@ -0,0 +1,94 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="hermes_infra",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=None)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
buildDocker = DockerComposeBuild(clone,
docker_repo_username="",
docker_repo_password="",
docker_compose_path="hermes_squid/stack-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
build_cache = True,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
# "user_group": "1000:137"
},
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=buildDocker,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="hermes_infra",
docker_compose_path="hermes_squid/stack-compose.yml",
)
# checkPing1 = UrlCheck(
# url="http://100.70.193.97:8088/api/status",
# expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
# )
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://hermes.rudolf.keenetic.link",
}
})
__all__ = ["hermes_infra"]
hermes_infra = Environment(
id="hermes_infra",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
# checkPing1,
logUrls
]
)
+27 -6
View File
@@ -4,7 +4,28 @@ from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="immich",
@@ -31,10 +52,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="poc/immich/stack-compose.yml",
)
checkPing = UrlCheck(
url="https://immich.rudolf.keenetic.link/api/server/ping",
expected={"res": "pong"},
)
#checkPing = UrlCheck(
# url="https://immich.rudolf.keenetic.link/api/server/ping",
# expected={"res": "pong"},
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://immich.rudolf.keenetic.link",
@@ -52,7 +73,7 @@ immich = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
]
)
+5 -1
View File
@@ -3,7 +3,11 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/reverseproxy.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/reverseproxy.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/reverseproxy"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/reverseproxy"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="torrserv_proxy",
+69 -5
View File
@@ -1,3 +1,4 @@
import hashlib
import logging
import os
import re
@@ -197,6 +198,32 @@ class DockerSwarmDeploy(AbstractStep[str]):
content = f.read()
content = re.sub(r'\$\{([^}]+)\}', lambda m: env.get(m.group(1), ""), content)
compose = yaml.safe_load(content)
version = env["version"]
if "configs" in compose:
sanitized_version = re.sub(r'[^a-zA-Z0-9_.-]', '_', version)
full_prefix = f"{self.stack_name}_"
def _config_full_name(base_name: str) -> str:
full = f"{full_prefix}{base_name}_{sanitized_version}"
if len(full) <= 64:
return f"{base_name}_{sanitized_version}"
return hashlib.sha256(full.encode("utf-8")).hexdigest()[:32]
config_names = {
name: _config_full_name(name)
for name in compose["configs"]
}
compose["configs"] = {
config_names.get(name, name): config
for name, config in compose["configs"].items()
}
for service in compose.get("services", {}).values():
for config in service.get("configs", []):
if isinstance(config, str):
config_index = service["configs"].index(config)
service["configs"][config_index] = config_names.get(config, config)
elif isinstance(config, dict) and "source" in config:
config["source"] = config_names.get(config["source"], config["source"])
if "services" in compose:
for svc in compose["services"].values():
if "build" in svc:
@@ -268,6 +295,7 @@ class DockerSwarmDeploy(AbstractStep[str]):
# Deploy stack to Docker Swarm
cmd = [
"docker", "stack", "deploy",
"--detach=false",
"--prune",
"-c", tmp_compose_path,
self.stack_name
@@ -279,12 +307,27 @@ class DockerSwarmDeploy(AbstractStep[str]):
swarmEnv = {k: v for k, v in dotenv_values(os.path.join(os.path.dirname(tmp_compose_path), ".env")).items()
if v is not None}
# merge_dicts(swarmEnv, env)
result = subprocess.run(cmd, capture_output=True, text=True)
# , cwd=os.path.dirname(tmp_compose_path), env=swarmEnv)
result = subprocess.run(cmd, capture_output=True, text=True, cwd=os.path.dirname(tmp_compose_path))
#, env=swarmEnv)
os.remove(tmp_compose_path)
if result.returncode != 0:
logger.error(f"Stack deploy failed: {result.stderr}")
raise RuntimeError(f"Stack deploy failed: {result.stderr}")
logger.info(f"docker stack deploy stdout: {result.stdout or '<empty>'}")
logger.info(f"docker stack deploy stderr: {result.stderr or '<empty>'}")
stderr_lower = result.stderr.lower()
docker_error_markers = (
"failed to ",
"error response from daemon",
"rpc error",
"denied",
"no such image",
)
if result.returncode != 0 or any(marker in stderr_lower for marker in docker_error_markers):
deploy_output = (
f"exit code: {result.returncode}\n"
f"stdout:\n{result.stdout or '<empty>'}\n"
f"stderr:\n{result.stderr or '<empty>'}"
)
logger.error(f"Stack deploy failed:\n{deploy_output}")
raise RuntimeError(f"Stack deploy failed:\n{deploy_output}")
logger.info(f"Stack deployed successfully: {result.stdout}")
# Check health of all deployed services
@@ -308,4 +351,25 @@ class DockerSwarmDeploy(AbstractStep[str]):
if not_running:
raise BaseException(f"Services not started yet: {', '.join(not_running)}")
version_mismatches = []
for svc_name, svc in expected_services.items():
full_svc_name = f"{self.stack_name}_{svc_name}"
services_list = client.services.list(filters={"name": full_svc_name})
if not services_list:
continue
service_labels = services_list[0].attrs.get("Spec", {}).get("TaskTemplate", {}).get("ContainerSpec", {}).get("Labels", {})
expected_version = svc.get("labels", {}).get("org.brencher.version")
actual_version = service_labels.get("org.brencher.version")
if expected_version != actual_version:
version_mismatches.append(
f"{full_svc_name} expected version {expected_version}, got {actual_version}"
)
update_status = services_list[0].attrs.get("UpdateStatus", {})
if update_status.get("State") in {"paused", "rollback_paused", "rollback_started"}:
version_mismatches.append(
f"{full_svc_name} Swarm update {update_status.get('State')}: {update_status.get('Message', '')}"
)
if version_mismatches:
raise RuntimeError(f"Deployment did not apply requested versions: {'; '.join(version_mismatches)}")
return f"Stack deployed successfully: {result.stdout}"
+125 -47
View File
@@ -16,11 +16,62 @@ logger = logging.getLogger(__name__)
class GitClone(AbstractStep[str]):
def __init__(self, url: str, repo_path: str | None = None,
branchNamePrefix: str = "", credEnvPrefix: str = "GIT",
def __init__(self,
url: str | List[str] | None = None,
remotes: List[Dict[str, str]] | None = None,
repo_path: str | None = None,
branchNamePrefix: str = "",
credEnvPrefix: str = "GIT",
**kwargs: Any):
super().__init__(**kwargs)
self.url = url
# `remotes` is the explicit form: a list of {"name": "...", "url": "..."}.
# `url` is kept as a shorthand: a single string becomes one remote named
# "origin"; a list of strings becomes remotes named "origin", "repo1",
# "repo2", ... (matches the behavior introduced by the "multiple remotes"
# change so existing positional-list callers keep working).
if remotes is not None and url is not None:
raise ValueError("Pass either 'url' or 'remotes', not both")
normalized: List[Dict[str, str]] = []
if remotes is not None:
seen: Set[str] = set()
for entry in remotes:
if not isinstance(entry, dict):
raise TypeError(
f"Each entry in 'remotes' must be a dict with 'name' and 'url', got {type(entry).__name__}"
)
name = entry.get("name")
url_value = entry.get("url")
if not name or not isinstance(name, str):
raise ValueError("Each remote requires a non-empty 'name' (string)")
if not url_value or not isinstance(url_value, str):
raise ValueError(f"Remote {name!r} requires a non-empty 'url' (string)")
if name in seen:
raise ValueError(f"Duplicate remote name: {name!r}")
prefix_value = entry.get("prefix")
if prefix_value is not None and not isinstance(prefix_value, str):
raise ValueError(
f"Remote {name!r} 'prefix' must be a string when set, got {type(prefix_value).__name__}"
)
seen.add(name)
normalized.append({"name": name, "url": url_value, "prefix": prefix_value or ""})
elif url is not None:
if isinstance(url, str):
raw = [url]
else:
raw = list(url)
for index, repository_url in enumerate(raw):
name = "origin" if index == 0 else f"repo{index}"
normalized.append({"name": name, "url": repository_url})
else:
raise ValueError("GitClone requires either 'url' or 'remotes'")
if len(normalized) == 0:
raise ValueError("At least one repository URL is required")
self.remotes = normalized
# Derived view used by repo_path keying and any introspection.
self.urls = [r["url"] for r in normalized]
self.repo_path = repo_path
self.branchNamePrefix = branchNamePrefix
self.credEnvPrefix = credEnvPrefix
@@ -36,33 +87,38 @@ class GitClone(AbstractStep[str]):
return url
def progress(self) -> str:
repo_url = self.url
self.repo_path = self.repo_path or os.path.join(tempfile.gettempdir(),
f"{self.env.id}_{hashlib.sha1(repo_url.encode()).hexdigest()[:5]}")
logger.info(f"Cloning repository {repo_url} to {self.repo_path}")
f"{self.env.id}_{hashlib.sha1('|'.join(self.urls).encode()).hexdigest()[:5]}")
logger.info(f"Cloning repositories {self.remotes} to {self.repo_path}")
os.makedirs(self.repo_path, exist_ok=True)
try:
if os.path.exists(os.path.join(self.repo_path, ".git")):
logger.info(f"Repository already cloned at {self.repo_path}, fetching updates.")
repo = git.Repo(self.repo_path)
if 'origin' not in repo.remotes or repo.remotes.origin.url != self._get_auth_git_url(repo_url):
repo.create_remote('origin', self._get_auth_git_url(repo_url))
result = repo.remotes.origin.fetch(prune=True)
if not result or any(fetch_info.flags & fetch_info.ERROR for fetch_info in result):
raise BaseException(f"Failed to fetch updates for {repo_url}")
else:
repo = git.Repo.init(self.repo_path)
repo.remotes.append(repo.create_remote(
'origin',
self._get_auth_git_url(repo_url)
))
if self.branchNamePrefix != "":
repo.config_writer().set_value('remote "origin"', "fetch",
f"+refs/heads/{self.branchNamePrefix}/*:refs/remotes/origin/{self.branchNamePrefix}/*").release()
repo.remotes.origin.fetch(prune=True)
if not os.path.exists(os.path.join(self.repo_path, ".git")):
raise BaseException(f"Failed to clone repository {repo_url} to {self.repo_path}")
for entry in self.remotes:
remote_name = entry["name"]
repository_url = entry["url"]
auth_url = self._get_auth_git_url(repository_url)
if remote_name in repo.remotes:
remote = repo.remotes[remote_name]
if remote.url != auth_url:
remote.set_url(auth_url)
else:
remote = repo.create_remote(remote_name, auth_url)
prefix = entry.get("prefix", "")
effective_prefix = prefix or (self.branchNamePrefix if remote_name == "origin" else "")
if effective_prefix:
with repo.config_writer() as config:
config.set_value(f'remote "{remote_name}"', "fetch",
f"+refs/heads/{effective_prefix}/*:refs/remotes/{remote_name}/{effective_prefix}/*")
result = remote.fetch(prune=True)
if not result or any(fetch_info.flags & fetch_info.ERROR for fetch_info in result):
raise BaseException(f"Failed to fetch updates for {repository_url}")
if not os.path.exists(os.path.join(self.repo_path, ".git")):
raise BaseException(f"Failed to clone repositories {self.remotes} to {self.repo_path}")
except BaseException as e:
logger.error(f"Error during git clone/fetch, removing directory {self.repo_path}: {str(e)}")
shutil.rmtree(self.repo_path)
@@ -73,21 +129,24 @@ class GitClone(AbstractStep[str]):
repo = git.Repo(self.repo_path)
result: Dict[str, List[Any]] = {}
for ref in repo.refs:
if ref.name.startswith('origin/') and not ref.name.startswith('origin/HEAD'):
branch_name = ref.name[len('origin/'):]
if not branch_name.startswith('auto/'): # Skip auto branches
result[branch_name] = []
cmt = [ref.commit]
for _ in range(10):
for commit in cmt:
result[branch_name].append({
'hexsha': commit.hexsha,
'author': commit.author.name,
'date': commit.committed_datetime.isoformat(),
'message': commit.message.strip()
})
if not ref.is_remote() or ref.name.endswith('/HEAD'):
continue
remote_name, branch_name = ref.name.split('/', 1)
if branch_name.startswith('auto/'): # Skip auto branches
continue
branch_key = branch_name if remote_name == 'origin' else ref.name
result[branch_key] = []
cmt = [ref.commit]
for _ in range(10):
for commit in cmt:
result[branch_key].append({
'hexsha': commit.hexsha,
'author': commit.author.name,
'date': commit.committed_datetime.isoformat(),
'message': commit.message.strip()
})
cmt = [p for c in cmt for p in c.parents]
cmt = [p for c in cmt for p in c.parents]
return result
@@ -112,6 +171,18 @@ def ensure_clean(repo: git.Repo) -> None:
raise BaseException(f"Changes in repo: U{repo.untracked_files}")
def _resolve_branch_ref(repo: git.Repo, branch_key: str, default_remote: str) -> str:
# Branch keys can be "branch", "origin/branch", "gitea_admin/branch", etc.
# If the key has a remote prefix that matches a known remote, use it
# verbatim; otherwise fall back to the primary configured remote so
# configs that don't name a remote `origin` still work.
if "/" in branch_key:
remote_name, _ = branch_key.split("/", 1)
if remote_name in repo.remotes:
return branch_key
return f"{default_remote}/{branch_key}"
class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
wd: GitClone
@@ -146,11 +217,19 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
def find_desired_commits(self, repo: git.Repo, branches: List[Tuple[str, str]]) -> Dict[Commit, str]:
# Extract commit ids for the selected branches
commit_ids: Dict[Commit, str] = {}
# Default remote: prefer the first remote GitClone was configured with
# (so non-"origin" configs like github/gitea_admin work without the
# caller having to qualify every branch key).
default_remote = "origin"
if isinstance(getattr(self.wd, "remotes", None), list) and self.wd.remotes:
primary = self.wd.remotes[0]["name"]
if primary in repo.remotes:
default_remote = primary
for branch_pair in branches:
try:
branch_name, desired_commit = branch_pair
if desired_commit == 'HEAD':
commit = repo.commit(f'origin/{branch_name}')
commit = repo.commit(_resolve_branch_ref(repo, branch_name, default_remote))
else:
commit = repo.commit(desired_commit)
commit_ids[commit] = branch_name
@@ -229,9 +308,10 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
version = '-'.join([x.hexsha[0:8] for x in sorted_commits])
for ref in repo.refs:
if ref.is_remote() and ref.commit == repo.head.commit and ref.name != 'origin/HEAD':
if ref.is_remote() and ref.commit == repo.head.commit and not ref.name.endswith('/HEAD'):
logger.info(f"Merge commit {ref.commit} corresponds to branch {ref}")
remote_branch_name = ref.name[len('origin/'):]
remote_name, branch_name = ref.name.split('/', 1)
remote_branch_name = branch_name if remote_name == 'origin' else ref.name
break
if self.push and remote_branch_name is None:
auto_branch_hash = hashlib.sha1(''.join([x.hexsha for x in sorted_commits]).encode()).hexdigest()
@@ -292,10 +372,9 @@ class GitUnmerge(AbstractStep[GitUnmergeResult]):
commits: List[Tuple[str, str]] = []
for v in version_parts:
commit = repo.commit(v)
# branches = [head.name for head in repo.heads if head.commit.hexsha == commit.hexsha]
branches = [ref.name for ref in repo.remotes.origin.refs if ref.commit.hexsha == commit.hexsha]
branches = [b[len('origin/'):] for b in branches if
b.startswith('origin/') and not b.startswith('origin/HEAD')]
branches = [ref.name for ref in repo.refs
if ref.is_remote() and not ref.name.endswith('/HEAD') and ref.commit.hexsha == commit.hexsha]
branches = [b[len('origin/'):] if b.startswith('origin/') else b for b in branches]
if len(branches) == 0 or branches[0].startswith('auto/'):
if childs is None:
@@ -304,10 +383,9 @@ class GitUnmerge(AbstractStep[GitUnmergeResult]):
while len(commitsSet) > 0:
current = commitsSet.pop()
for child in childs.get(current, []):
branches = [ref.name for ref in repo.remotes.origin.refs if
ref.commit.hexsha == child.hexsha]
branches = [b[len('origin/'):] for b in branches if
b.startswith('origin/') and not b.startswith('origin/HEAD')]
branches = [ref.name for ref in repo.refs
if ref.is_remote() and not ref.name.endswith('/HEAD') and ref.commit.hexsha == child.hexsha]
branches = [b[len('origin/'):] if b.startswith('origin/') else b for b in branches]
if len(branches) > 0:
commitsSet = set()
+5 -2
View File
@@ -42,8 +42,11 @@ class SharedStateHolderInMemory(AbstractStep[SharedState], SharedStateHolder):
def progress(self) -> SharedState:
if len(self.state.branches) == 0:
if self.unmerge:
unmerge = self.unmerge.progress()
self.state = replace(self.state, branches=unmerge.branches, token=uuid.uuid4().hex)
try:
unmerge = self.unmerge.progress()
self.state = replace(self.state, branches=unmerge.branches, token=uuid.uuid4().hex)
except BaseException as e:
logger.error("Unmerge failed: %s", e)
return self.state
+76 -33
View File
@@ -10,12 +10,10 @@ from fastapi import FastAPI, Response, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse
from app import App
from enironment import SharedStateConflictError, get_step
from app import App
from utils import custom_json_dumps
from processing import reset_caches
from secondary import SecondaryManager
from steps.step import CachingStep
logger = logging.getLogger(__name__)
@@ -47,12 +45,14 @@ class WebApp:
self.port = port
self.ws_connections: Dict[WebSocket, Dict[str, Any]] = {}
self.ws_lock = asyncio.Lock()
self._event_loop: Optional[asyncio.AbstractEventLoop] = None
self._server: Any = None
self.secondaryManager: Optional[SecondaryManager] = None
@asynccontextmanager
async def lifespan(fastapi_app: FastAPI) -> AsyncIterator[None]:
self._event_loop = asyncio.get_event_loop()
self._event_loop = asyncio.get_running_loop()
self.secondaryManager = SecondaryManager(
urls=os.getenv("SECONDARY_BRENCHER", ""),
on_update=lambda: self.broadcast_all(),
@@ -94,10 +94,20 @@ class WebApp:
merged = _merge_dicts(merged, connector.branches)
return merged
# --- Async scheduling from sync threads ---
# --- Async scheduling from sync callbacks ---
def _schedule_async(self, coro: Any) -> None:
if self._event_loop is not None and self._event_loop.is_running():
if self._event_loop is None or not self._event_loop.is_running():
logger.warning("Dropping scheduled coroutine because the web event loop is not running")
coro.close()
return
try:
running_loop = asyncio.get_running_loop()
except RuntimeError:
running_loop = None
if running_loop is self._event_loop:
self._event_loop.create_task(coro)
else:
asyncio.run_coroutine_threadsafe(coro, self._event_loop)
# --- Broadcasting ---
@@ -107,18 +117,19 @@ class WebApp:
disconnected = set()
message = custom_json_dumps({event: data})
for websocket in self.ws_connections.keys():
if self.ws_connections[websocket].get(event) == message:
continue
try:
await websocket.send_text(message)
self.ws_connections[websocket][event] = message
except Exception as e:
logger.error(f"Error sending to websocket: {e}")
disconnected.add(websocket)
async with self.ws_lock:
for websocket in list(self.ws_connections.keys()):
if self.ws_connections[websocket].get(event) == message:
continue
try:
await websocket.send_text(message)
self.ws_connections[websocket][event] = message
except Exception as e:
logger.error(f"Error sending to websocket: {e}")
disconnected.add(websocket)
for ws in disconnected:
self.ws_connections.pop(ws, None)
for ws in disconnected:
self.ws_connections.pop(ws, None)
async def broadcast_all(self) -> None:
await self.broadcast("branches", self.get_global_branches_to_emit())
@@ -129,22 +140,28 @@ class WebApp:
async def broadcast_environments(self, data: Any) -> None:
await self.broadcast("environments", data)
await self.broadcast("version", self.get_version())
async def broadcast_error(self, data: Any) -> None:
await self.broadcast("error", data)
def emit_envs(self) -> None:
"""Sync callback used by App's processing thread to push updates to clients."""
"""Sync callback used by App's processing loop to push updates to clients."""
self._schedule_async(self.broadcast_branches(self.get_global_branches_to_emit()))
self._schedule_async(self.broadcast_environments(self.get_global_envs_to_emit()))
# --- Routes ---
def get_version(self) -> str:
return os.getenv('VERSION', 'unknown') or 'unknown'
async def serve_index(self) -> FileResponse:
return FileResponse(os.path.join(FRONTEND_DIR, 'index.html'))
async def serve_state(self) -> Response:
return Response(content=custom_json_dumps(self.get_global_envs_to_emit()), media_type="application/json")
state = self.get_global_envs_to_emit()
state['version'] = self.get_version()
return Response(content=custom_json_dumps(state), media_type="application/json")
async def serve_branches_route(self) -> Response:
return Response(content=custom_json_dumps(self.get_global_branches_to_emit()), media_type="application/json")
@@ -157,16 +174,26 @@ class WebApp:
async def websocket_endpoint(self, websocket: WebSocket) -> None:
await websocket.accept()
self.ws_connections[websocket] = {}
# Compute payloads first so the dict registration below is a single
# atomic update under ws_lock — avoids the window where the key
# exists but the inner dict is empty, which races with broadcast()
# cleanup and crashes with KeyError at self.ws_connections[ws][...] = ...
branches_payload = custom_json_dumps({"branches": self.get_global_branches_to_emit()})
envs_payload = custom_json_dumps({"environments": self.get_global_envs_to_emit()})
version_payload = custom_json_dumps({"version": self.get_version()})
async with self.ws_lock:
self.ws_connections[websocket] = {
"version": version_payload,
"branches": branches_payload,
"environments": envs_payload,
}
try:
# Send initial state on connect and record it so duplicate broadcasts are suppressed
branches_payload = custom_json_dumps({"branches": self.get_global_branches_to_emit()})
envs_payload = custom_json_dumps({"environments": self.get_global_envs_to_emit()})
await websocket.send_text(version_payload)
await websocket.send_text(branches_payload)
self.ws_connections[websocket]["branches"] = branches_payload
await websocket.send_text(envs_payload)
self.ws_connections[websocket]["environments"] = envs_payload
while True:
data = await websocket.receive_text()
@@ -179,7 +206,7 @@ class WebApp:
await self.secondaryManager.send({"update": update_data})
id = update_data.get('id', '')
if id == '':
reset_caches(list(self.core.environments.values()))
self.core.request_reset()
elif id not in self.core.environments.keys() and id not in {j for it in self.secondaryManager or [] for j in it.environments.keys()}:
logger.warning(f"Received update for unknown environment id {id}")
continue
@@ -210,22 +237,38 @@ class WebApp:
continue
if env:
p = get_step(env.pipeline, type(env.state))
if isinstance(p, CachingStep):
p.reset()
self.core.request_reset()
logger.info(f"Updated environment {env.id} branches to {update_data.get('branches')}, dry={update_data.get('dry')}")
self.core.environment_update_event.set()
self.core.notify_environment_update()
await self.broadcast_environments(self.get_global_envs_to_emit())
except WebSocketDisconnect:
self.ws_connections.pop(websocket, None)
async with self.ws_lock:
self.ws_connections.pop(websocket, None)
except Exception as e:
logger.error(f"WebSocket error: {e}:{traceback.format_exc()}")
await self.broadcast_error({'message': f'{e}:{traceback.format_exc()}'})
# --- Run ---
def start(self) -> None:
async def start_async(self) -> None:
import uvicorn
uvicorn.run(self.app, host='0.0.0.0', port=self.port, log_level='info')
config = uvicorn.Config(self.app, host='0.0.0.0', port=self.port, log_level='info')
self._server = uvicorn.Server(config)
try:
await self._server.serve()
except asyncio.CancelledError:
self._server.should_exit = True
raise
def start(self) -> None:
asyncio.run(self.start_async())
def stop(self) -> None:
if self._server is None:
return
if self._event_loop is not None and self._event_loop.is_running():
self._event_loop.call_soon_threadsafe(lambda: setattr(self._server, "should_exit", True))
else:
self._server.should_exit = True
+2 -1
View File
@@ -16,7 +16,8 @@ services:
- PYTHONUNBUFFERED=1
- VIRTUAL_HOST=brencher.rudolf.keenetic.link
- LETSENCRYPT_HOST=brencher.rudolf.keenetic.link
- PROFILES=brencher2,torrserv_proxy,immich,registry,gmail_mcp
- VERSION=${version}
- PROFILES=brencher2,torrserv_proxy,immich,gmail_mcp,hermes,hermes_infra,gitea
- SECONDARY_BRENCHER=http://host.docker.internal:5002
deploy:
replicas: 1
+3
View File
@@ -13,6 +13,7 @@ const closeStatus = document.getElementById('close-status');
const refreshBranchesBtn = document.getElementById('refresh-branches');
const applyChangesBtn = document.getElementById('apply-changes');
const branchFilter = document.getElementById('branch-filter');
const appVersion = document.getElementById('app-version');
// Branches list (flattened) for UI rendering: [{ envId, envName, branch, commits: [] }]
let branches = [];
@@ -477,6 +478,8 @@ function setupWebSockets() {
filterBranches();
renderJobs();
showStatus('Environments updated.');
} else if ('version' in message) {
appVersion.textContent = `Version: ${message.version || 'unknown'}`;
} else if ('error' in message) {
console.error('Error from server:', message.error);
if (message.error && message.error.code === 'BRANCH_STATE_CONFLICT' && message.error.envId) {
+1
View File
@@ -10,6 +10,7 @@
<div class="container">
<header>
<h1>Brencher - Branch Merger</h1>
<p class="version" id="app-version">Version: loading...</p>
<div id="status-bar" class="status-bar hidden">
<span id="status-message"></span>
<button id="close-status">×</button>
+7
View File
@@ -15,6 +15,13 @@ body {
padding: 32px;
}
.version {
color: #6c757d;
font-family: monospace;
font-size: 0.85rem;
margin-top: -12px;
}
.primary-button {
background: #007bff;
color: #fff;
+4
View File
@@ -11,3 +11,7 @@ types-PyYAML
types-requests
types-docker
testcontainers
+20
View File
@@ -1,8 +1,28 @@
import subprocess
from pathlib import Path
from enironment import Environment
from steps.docker_plain import DockerImageBuild, DockerContainerCheck, DockerContainerDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
REPO_ROOT = Path(__file__).resolve().parents[2]
def _git_output(*args: str) -> str:
try:
return subprocess.check_output(
["git", "-C", str(REPO_ROOT), *args],
text=True,
stderr=subprocess.STDOUT,
).strip()
except subprocess.CalledProcessError as exc:
raise RuntimeError(f"Unable to read git metadata for nginx test config: {' '.join(args)}\n{exc.output}") from exc
CURRENT_BRANCH = _git_output("rev-parse", "--abbrev-ref", "HEAD")
CURRENT_COMMIT = _git_output("rev-parse", "HEAD")
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
container_check = DockerContainerCheck(
+52
View File
@@ -0,0 +1,52 @@
import asyncio
import sys
import types
from typing import Callable
import pytest
from enironment import Environment
import processing
from app import App
class TestAppAsyncio:
@pytest.mark.asyncio
async def test_run_web_async_stops_processing_when_server_exits(self, monkeypatch: pytest.MonkeyPatch) -> None:
process_calls = 0
emit_calls = 0
processing_started = asyncio.Event()
loop = asyncio.get_running_loop()
web_app_args: dict[str, object] = {}
def fake_process_all_jobs(environments: list[Environment], onupdate: Callable[[], None]) -> bool:
nonlocal process_calls
process_calls += 1
assert environments == []
onupdate()
loop.call_soon_threadsafe(processing_started.set)
return False
class FakeWebApp:
def __init__(self, core: App, port: int) -> None:
web_app_args["core"] = core
web_app_args["port"] = port
def emit_envs(self) -> None:
nonlocal emit_calls
emit_calls += 1
async def start_async(self) -> None:
await asyncio.wait_for(processing_started.wait(), timeout=1)
monkeypatch.setattr(processing, "process_all_jobs", fake_process_all_jobs)
monkeypatch.setitem(sys.modules, "web", types.SimpleNamespace(WebApp=FakeWebApp))
app = App({})
await asyncio.wait_for(app.runWebAsync(5001), timeout=1)
assert process_calls >= 1
assert emit_calls >= 1
assert web_app_args == {"core": app, "port": 5001}
assert app.emit_callback is None
+21 -21
View File
@@ -1,7 +1,7 @@
import asyncio
import json
import logging
import threading
from contextlib import suppress
from typing import Any, Generator
import docker
@@ -36,30 +36,30 @@ class TestDockerContainer:
logger.info(f"Starting")
app = App({"brencher_local1": brencher_local1.brencher_local1})
# app.runWeb(5001)
processing = threading.Thread(target=lambda: app.runWeb(5001), daemon=True)
processing.start()
processing = asyncio.create_task(app.runWebAsync(5001))
try:
eventually(
lambda: assert_equal(requests.get("http://localhost:5001/state", timeout=5000).status_code, 200, "Server did not respond with 200 OK within timeout"),
20.0,
1.0,
)
# await asyncio.sleep(5000)
state_data = requests.get("http://localhost:5001/state", timeout=5000).json()
logger.info(f"Application state: {json.dumps(state_data, indent=2)}")
eventually(
lambda: assert_equal(requests.get("http://localhost:5001/state", timeout=5000).status_code, 200, "Server did not respond with 200 OK within timeout"),
20.0,
1.0,
)
branches_data = requests.get("http://localhost:5001/branches", timeout=5000).json()
logger.info(f"Application branches: {json.dumps(branches_data, indent=2)}")
assert "brencher_local1" in state_data, "Missing 'brencher_local1' field in response"
assert len(state_data["brencher_local1"]["pipeline"]) > 0, "Steps defined for 'brencher_local1' are empty"
state_data = requests.get("http://localhost:5001/state", timeout=5000).json()
logger.info(f"Application state: {json.dumps(state_data, indent=2)}")
branches_data = requests.get("http://localhost:5001/branches", timeout=5000).json()
logger.info(f"Application branches: {json.dumps(branches_data, indent=2)}")
assert "brencher_local1" in state_data, "Missing 'brencher_local1' field in response"
assert len(state_data["brencher_local1"]["pipeline"]) > 0, "Steps defined for 'brencher_local1' are empty"
assert "brencher_local1" in branches_data, "Missing 'brencher_local1' field in response"
assert len(branches_data["brencher_local1"]) > 0, "No branches found for 'brencher_local1'"
assert "brencher_local1" in branches_data, "Missing 'brencher_local1' field in response"
assert len(branches_data["brencher_local1"]) > 0, "No branches found for 'brencher_local1'"
finally:
processing.cancel()
with suppress(asyncio.CancelledError):
await processing
assert processing.cancelled()
# TODO Verify DockerContainerDeploy step status is running and has correct image and ports
# {
+53 -41
View File
@@ -78,55 +78,67 @@ class TestDryRunPlaywright:
# Start app with nginx_local1 with main branch pre-selected and dry mode on.
env = nginx.test_local1
env.state.set_branches([("test/main", "HEAD")])
env.state.set_branches([(nginx.CURRENT_BRANCH, nginx.CURRENT_COMMIT)])
env.state.set_dry(True)
app = App({env.id: env})
_server_thread = threading.Thread(target=lambda: app.runWeb(APP_PORT), daemon=True)
_server_thread.start()
try:
# Wait for server to be up
eventually(lambda: assert_equal(requests.get(f"{APP_URL}/state", timeout=5).status_code, 200, "Server did not respond with 200 OK within timeout"))
logger.info("Server is up")
# Wait for server to be up
eventually(lambda: assert_equal(requests.get(f"{APP_URL}/state", timeout=5).status_code, 200, "Server did not respond with 200 OK within timeout"))
logger.info("Server is up")
# Wait for the pipeline to stabilize with dry=True:
# DockerContainerDeploy should return status="dry-run"
with brencher_page(APP_URL, env.id) as page:
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] == "SharedStateHolderInMemory"
)["status"]['dry'], True, "Pipeline step status is not 'running'")),
)
# Wait for the pipeline to stabilize with dry=True:
# DockerContainerDeploy should return status="dry-run"
with brencher_page(APP_URL, env.id) as page:
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] == "SharedStateHolderInMemory"
)["status"]['dry'], True, "Pipeline step status is not 'running'")),
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "dry-run")
logger.info("Pipeline is in dry-run mode")
# page.verify_pipeline_step_status("DockerContainerDeploy", "dry-run")
logger.info("Pipeline is in dry-run mode")
# Verify no real container was created
docker_client = docker.from_env()
real_containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(real_containers) == 0, "Container should NOT exist during dry run"
# Verify no real container was created
docker_client = docker.from_env()
real_containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(real_containers) == 0, "Container should NOT exist during dry run"
# Disable dry run from the UI.
page.verify_dry_run_on()
page.set_dry_run_off()
page.verify_dry_run_off()
# Disable dry run from the UI.
page.verify_dry_run_on()
page.set_dry_run_off()
page.verify_dry_run_off()
page.click_refresh() # TODO Remove it
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] =="SharedStateHolderInMemory"
)["status"]['dry'], False, "Pipeline step status is not 'running'")),
)
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] =="SharedStateHolderInMemory"
)["status"]['dry'], False, "Pipeline step status is not 'running'")),
)
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal({p["name"] for p in s[env.id]["pipeline"] if bool(p['is_running'])}, set(), "Pipeline still running")),
timeout=20,
interval=1.0,
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "running")
logger.info("Container deployed and running after dry run disabled")
# Wait for the real container to be created and started.
eventually(
lambda: _assert_container_running(docker_client),
# Allow time for a cold image build and first container start on CI runners.
timeout=180,
interval=5.0,
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "running")
logger.info("Container deployed and running after dry run disabled")
# Final sanity: real container must exist and be running
containers = docker_client.containers.list(filters={"name": CONTAINER_NAME})
assert len(containers) == 1, "Container should exist and be running after deploy"
assert containers[0].status == "running"
# Final sanity: real container must exist and be running
containers = docker_client.containers.list(filters={"name": CONTAINER_NAME})
assert len(containers) == 1, "Container should exist and be running after deploy"
assert containers[0].status == "running"
finally:
app.stop()
server_thread.join(timeout=30)
if server_thread.is_alive():
logger.warning("App server thread did not stop within 30 seconds")
def _assert_container_running(docker_client: docker.DockerClient) -> None:
containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(containers) > 0, "Container should exist after deploy"
assert_equal({container.status for container in containers}, {"running"}, "Container should be running after deploy")
+184 -1
View File
@@ -5,10 +5,12 @@ These tests run in a Docker-like isolated environment to avoid harming the host
They simulate remote and local git repositories and test various merge scenarios.
"""
import os
import shutil
import tempfile
import git
import pytest
from steps.git import CheckoutAndMergeResult, GitUnmergeResult
from steps.git import CheckoutAndMergeResult, GitClone, GitUnmergeResult
from .test_remote_repo import RemoteRepoHelper
@@ -16,6 +18,187 @@ from .test_remote_repo import RemoteRepoHelper
class TestGitIntegration:
"""Integration tests for git operations"""
def test_git_clone_multiple_urls(self, repo_helper: RemoteRepoHelper) -> None:
repo_helper.create_commit(repo_helper.repo, "master", "master", "README.md", "content", "Initial commit")
second_url = f"file://{repo_helper.remote_dir}"
clone = GitClone(url=[repo_helper.remote_dir, second_url])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
assert {remote.name for remote in repo.remotes} == {"origin", "repo1"}
assert repo.remotes.origin.url == repo_helper.remote_dir
assert repo.remotes.repo1.url == second_url
assert set(clone.get_branches()) == {"master", "repo1/master"}
def test_git_clone_named_remotes(self, repo_helper: RemoteRepoHelper) -> None:
# Two real local repos served as separate "remotes". The user controls
# their names explicitly via the `remotes=[{name,url}]` API.
def fresh_repo(prefix: str, message: str) -> str:
path = tempfile.mkdtemp(prefix=prefix)
repo = git.Repo.init(path, bare=False)
with repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
repo_helper.create_commit(repo, "master", "master", "README.md", "content", message)
return path
github_dir = fresh_repo("test_named_github_", "github initial")
gitea_dir = fresh_repo("test_named_gitea_", "gitea initial")
try:
clone = GitClone(remotes=[
{"name": "github", "url": github_dir},
{"name": "gitea_admin", "url": gitea_dir},
])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
# User-chosen names appear exactly; the auto-generated "origin" /
# "repo1" names from the positional-list form do NOT leak in.
assert {r.name for r in repo.remotes} == {"github", "gitea_admin"}
assert "origin" not in repo.remotes
assert "repo1" not in repo.remotes
assert repo.remotes.github.url == github_dir
assert repo.remotes.gitea_admin.url == gitea_dir
# `get_branches` keeps the <remote>/<branch> shape for non-origin
# names; the user gets the full provenance in the key.
assert set(clone.get_branches()) == {"github/master", "gitea_admin/master"}
finally:
shutil.rmtree(github_dir, ignore_errors=True)
shutil.rmtree(gitea_dir, ignore_errors=True)
def test_git_clone_remotes_validation(self) -> None:
# Empty list: rejected (would produce an unusable clone).
with pytest.raises(ValueError, match="At least one repository URL is required"):
GitClone(remotes=[])
# Missing / empty / non-string name or url: rejected.
with pytest.raises(ValueError, match="non-empty 'name'"):
GitClone(remotes=[{"url": "git@example.com:foo/bar.git"}])
with pytest.raises(ValueError, match="non-empty 'name'"):
GitClone(remotes=[{"name": "", "url": "git@example.com:foo/bar.git"}])
with pytest.raises(ValueError, match="non-empty 'url'"):
GitClone(remotes=[{"name": "github"}])
with pytest.raises(ValueError, match="non-empty 'url'"):
GitClone(remotes=[{"name": "github", "url": ""}])
# Duplicate name: rejected.
with pytest.raises(ValueError, match="Duplicate remote name"):
GitClone(remotes=[
{"name": "github", "url": "git@example.com:a/b.git"},
{"name": "github", "url": "git@example.com:c/d.git"},
])
# Passing both `url` and `remotes`: rejected.
with pytest.raises(ValueError, match="not both"):
GitClone(url="git@example.com:a/b.git", remotes=[{"name": "github", "url": "git@example.com:a/b.git"}])
# Passing neither: rejected.
with pytest.raises(ValueError, match="requires either 'url' or 'remotes'"):
GitClone()
def test_git_clone_remotes_per_remote_prefix(self, repo_helper: RemoteRepoHelper) -> None:
# Per-remote `prefix` filters fetch refspec so only matching branches are visible.
# Two repos, two branches each (one matching the prefix, one not). Only the
# matching branch should appear in get_branches() for each remote.
def fresh_repo(prefix: str) -> str:
path = tempfile.mkdtemp(prefix=prefix)
repo = git.Repo.init(path, bare=False)
with repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
# Branches literally named `ansible/...` to match the prefix filter.
# create_commit creates an initial master, then branches off it.
repo_helper.create_commit(repo, "master", "master", "README.md", "content", f"{prefix} init")
repo_helper.create_commit(repo, "master", "ansible/master", "AN.md", "ansible", f"{prefix} ansible")
repo_helper.create_commit(repo, "master", "ansible/fix", "FIX.md", "fix", f"{prefix} fix")
return path
github_dir = fresh_repo("test_prefix_github_")
gitea_dir = fresh_repo("test_prefix_gitea_")
try:
clone = GitClone(remotes=[
{"name": "github", "url": github_dir, "prefix": "ansible"},
{"name": "gitea_admin", "url": gitea_dir, "prefix": "ansible"},
])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
branches = set(clone.get_branches())
# Per-remote refspec was applied: only ansible/* branches fetched.
assert "github/ansible/master" in branches, f"missing: {branches}"
assert "github/ansible/fix" in branches, f"missing: {branches}"
assert "gitea_admin/ansible/master" in branches, f"missing: {branches}"
assert "gitea_admin/ansible/fix" in branches, f"missing: {branches}"
# `master` exists upstream but is NOT visible (filtered by prefix).
assert "github/master" not in branches, f"unwanted: {branches}"
assert "gitea_admin/master" not in branches, f"unwanted: {branches}"
finally:
shutil.rmtree(github_dir, ignore_errors=True)
shutil.rmtree(gitea_dir, ignore_errors=True)
def test_git_clone_remotes_mixed_prefix(self, repo_helper: RemoteRepoHelper) -> None:
# Some remotes with prefix, others without. Each behaves independently:
# prefixed remotes only see matching branches, unprefixed see all.
def fresh_repo(prefix: str) -> str:
path = tempfile.mkdtemp(prefix=prefix)
repo = git.Repo.init(path, bare=False)
with repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
repo_helper.create_commit(repo, "master", "master", "README.md", "content", f"{prefix} init")
repo_helper.create_commit(repo, "master", "ansible/master", "AN.md", "ansible", f"{prefix} ansible")
repo_helper.create_commit(repo, "master", "feature_y", "OTHER.md", "other", f"{prefix} feature")
return path
github_dir = fresh_repo("test_mixed_github_")
gitea_dir = fresh_repo("test_mixed_gitea_")
try:
clone = GitClone(remotes=[
{"name": "github", "url": github_dir, "prefix": "ansible"},
{"name": "gitea_admin", "url": gitea_dir},
])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
branches = set(clone.get_branches())
# Prefixed: only ansible/master.
assert "github/ansible/master" in branches, f"missing: {branches}"
assert "github/feature_y" not in branches, f"unwanted: {branches}"
assert "github/master" not in branches, f"unwanted: {branches}"
# Unprefixed: all branches visible.
assert "gitea_admin/ansible/master" in branches, f"missing: {branches}"
assert "gitea_admin/feature_y" in branches, f"missing: {branches}"
assert "gitea_admin/master" in branches, f"missing: {branches}"
finally:
shutil.rmtree(github_dir, ignore_errors=True)
shutil.rmtree(gitea_dir, ignore_errors=True)
def test_git_clone_remotes_prefix_validation(self) -> None:
# `prefix` must be a string when set; non-string raises ValueError.
with pytest.raises(ValueError, match="'prefix' must be a string"):
GitClone(remotes=[
{"name": "github", "url": "git@example.com:a/b.git", "prefix": 123},
])
with pytest.raises(ValueError, match="'prefix' must be a string"):
GitClone(remotes=[
{"name": "github", "url": "git@example.com:a/b.git", "prefix": ["ansible"]},
])
# Empty string / None: accepted, treated as no prefix.
c1 = GitClone(remotes=[{"name": "github", "url": "git@example.com:a/b.git", "prefix": ""}])
assert c1.remotes[0]["prefix"] == ""
c2 = GitClone(remotes=[{"name": "github", "url": "git@example.com:a/b.git"}])
assert c2.remotes[0]["prefix"] == ""
def test_checkout_merged_one_branch(self, repo_helper: RemoteRepoHelper) -> None:
# Create main branch with initial commit
commit1 = repo_helper.create_commit(repo_helper.repo, "master", "master", "file1.txt", "content1",
+188
View File
@@ -0,0 +1,188 @@
"""
Regression tests for the WebSocket ws_connections race fixed in PR #42.
Background
----------
Before the fix, ``WebApp.ws_connections`` (a Dict[WebSocket, Dict[str, Any]])
was modified from two uncoordinated coroutines: ``websocket_endpoint`` and
``broadcast()``. Adding the key, then awaiting ``send_text``, then assigning
``self.ws_connections[websocket]["branches"] = ...`` left a window where the
key existed but the inner dict was empty. A concurrent ``broadcast()`` loop
could ``pop(ws)`` the same key, and the next assignment crashed with::
File "/app/backend/web.py", line 184, in websocket_endpoint
self.ws_connections[websocket]["branches"] = branches_payload
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: <starlette.websockets.WebSocket object at 0x...>
The fix serialises every mutation of ``ws_connections`` under an
``asyncio.Lock`` and registers the websocket with all three payloads in a
single atomic assignment before the first ``send_text`` await.
These tests verify the lock contract on a real ``WebApp`` instance with
fake websocket objects that simulate the production timing.
"""
import asyncio
import sys
from typing import Any, Dict, List
import pytest
# Match the project's import layout: backend modules are at top level.
sys.path.insert(0, "backend")
from web import WebApp
class FakeWebSocket:
"""Minimal stand-in for starlette.websockets.WebSocket.
- ``send_delay`` lets us widen the race window between accept and dict
writes, mimicking slow clients.
- ``receive_text`` raises ``WebSocketDisconnect`` so the endpoint exits
cleanly after one round-trip.
"""
def __init__(self, send_delay: float = 0.0) -> None:
self.sent: List[str] = []
self.accepted = False
self.send_delay = send_delay
async def accept(self) -> None:
self.accepted = True
async def send_text(self, message: str) -> None:
if self.send_delay:
await asyncio.sleep(self.send_delay)
self.sent.append(message)
async def receive_text(self) -> str:
from starlette.websockets import WebSocketDisconnect
raise WebSocketDisconnect()
class FakeCore:
"""Minimal stand-in for ``app.App`` — only the methods WebApp touches."""
def __init__(self) -> None:
self.local_envs: Dict[str, Any] = {}
def get_local_envs_to_emit(self) -> Dict[str, Any]:
return self.local_envs
def get_local_branches_to_emit(self) -> Dict[str, Any]:
return {}
def _make_web_app() -> WebApp:
return WebApp(core=FakeCore(), port=0) # type: ignore[arg-type]
@pytest.mark.asyncio
async def test_websocket_endpoint_registers_atomically_under_lock() -> None:
"""websocket_endpoint must register the ws with all three payloads in a
single dict assignment under ``ws_lock``, BEFORE any ``send_text``.
The lock and the atomic dict write are the contract; if either is
missing, this test fails with KeyError or with the dict missing keys.
"""
web_app = _make_web_app()
ws = FakeWebSocket(send_delay=0.01) # widen the race window
async def concurrent_broadcast() -> None:
# Try to interleave several broadcasts while the endpoint is
# awaiting send_text. Pre-fix, this would race with the dict
# updates and either crash or drop the connection.
for _ in range(5):
await asyncio.sleep(0.005)
await web_app.broadcast("branches", {"x": 1})
task = asyncio.create_task(concurrent_broadcast())
await web_app.websocket_endpoint(ws) # type: ignore[arg-type]
await task
# Endpoint must have accepted the connection
assert ws.accepted
# Three initial send_text calls (version, branches, environments)
assert len(ws.sent) >= 3
# After endpoint exits cleanly (WebSocketDisconnect), the entry is gone
assert ws not in web_app.ws_connections
@pytest.mark.asyncio
async def test_broadcast_and_concurrent_register_dont_crash() -> None:
"""``broadcast()`` and a concurrent ``websocket_endpoint`` registration
must serialise cleanly under the lock. Pre-fix this could raise
``RuntimeError: dictionary changed size during iteration`` or
``KeyError`` when the outer key was popped between accept and write.
"""
web_app = _make_web_app()
n = 20
sockets = [FakeWebSocket(send_delay=0.002) for _ in range(n)]
async def register(s: FakeWebSocket) -> None:
await web_app.websocket_endpoint(s) # type: ignore[arg-type]
async def broadcaster() -> None:
for _ in range(10):
await web_app.broadcast("environments", {"k": "v"})
await asyncio.sleep(0)
# All endpoints + broadcasts run concurrently. Pre-fix this would crash
# with KeyError or RuntimeError on most runs.
await asyncio.gather(
*(register(s) for s in sockets),
broadcaster(),
)
# All endpoints should have cleaned up after themselves.
async with web_app.ws_lock:
leftover = list(web_app.ws_connections.keys())
assert leftover == [], f"expected ws_connections to drain, got {leftover}"
@pytest.mark.asyncio
async def test_ws_lock_exists_and_is_asyncio_lock() -> None:
"""Sanity check on the lock attribute — guards against accidental
removal in future refactors.
"""
import asyncio as _asyncio
web_app = _make_web_app()
assert isinstance(web_app.ws_lock, _asyncio.Lock)
@pytest.mark.asyncio
async def test_broadcast_skips_already_sent_payload() -> None:
"""broadcast() must not re-send a payload it has already sent to a
connection (used for de-dup). This is the second contract of the
function (the first being serialised mutation); ensure the lock didn't
break the de-dup check.
"""
from web import custom_json_dumps
web_app = _make_web_app()
ws = FakeWebSocket()
# Pre-register a websocket with the EXACT JSON payload that broadcast()
# will produce. ``broadcast(event, data)`` JSON-encodes
# ``{event: data}`` into the message and compares against the stored
# value, so we need to mirror that here.
payload = {"branches": "msg-1"}
web_app.ws_connections[ws] = {"branches": custom_json_dumps(payload)}
# Two broadcasts with the same payload — second one must be de-duped
# against the stored value (the de-dup check on line 120:
# ``if self.ws_connections[websocket].get(event) == message: continue``).
await web_app.broadcast("branches", payload)
# After the first call, the de-dup is in place — a second identical
# broadcast must not reach send_text.
sent_before = len(ws.sent)
await web_app.broadcast("branches", payload)
assert len(ws.sent) == sent_before, (
f"de-dup broken: second broadcast sent {len(ws.sent) - sent_before} extra message(s)"
)
# Cleanup
async with web_app.ws_lock:
web_app.ws_connections.pop(ws, None)