Compare commits

...
Author SHA1 Message Date
hermes 7fc6ad7438 ci: retrigger to test merge_main workflow with PUSH_TOKEN 2026-09-04 11:08:33 +00:00
hermes 43b5afaf94 ci: retrigger after comment cleanup
Python package / build (3.12.3) (pull_request) Successful in 4m17s
Python package / build (3.13) (pull_request) Successful in 2m15s
Python package / build (3.14) (pull_request) Successful in 2m28s
2026-09-04 07:31:40 +00:00
admin e3bc8a3212 Merge branch 'main' into feat/optimize-ci-build-time
Python package / build (3.12.3) (pull_request) Successful in 10m47s
Python package / build (3.13) (pull_request) Successful in 10m50s
Python package / build (3.14) (pull_request) Successful in 10m53s
2026-09-04 08:30:58 +02:00
hermes 0cb62d233a ci: trim explanatory comments from python-package.yml
Python package / build (3.12.3) (pull_request) Successful in 2m17s
Python package / build (3.13) (pull_request) Failing after 11m17s
Python package / build (3.14) (pull_request) Canceled after 11m58s
Drop the CI-history and per-option narration comments; keep only what is
needed for the workflow to be readable (job names, step names). Mirrors
the repo's compose/env/entrypoint-with-no-explanatory-comments style.
2026-09-04 06:15:05 +00:00
hermes ac682c1a92 ci: retrigger after run #58 success
Python package / build (3.12.3) (pull_request) Successful in 2m10s
Python package / build (3.13) (pull_request) Successful in 2m12s
Python package / build (3.14) (pull_request) Successful in 2m22s
2026-09-03 21:43:42 +00:00
hermes 0bbb3c4c26 ci: retrigger after runner cache register fix
Python package / build (3.12.3) (pull_request) Successful in 3m45s
Python package / build (3.13) (pull_request) Successful in 2m11s
Python package / build (3.14) (pull_request) Successful in 2m18s
2026-09-03 21:17:35 +00:00
hermes c6af53ec7e ci: empty commit to retrigger cache verification run
Python package / build (3.12.3) (pull_request) Canceled after 0s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
2026-09-03 20:42:40 +00:00
hermesandadmin e690f6a50c feat(configs): point torrserv_proxy at gitea admin/hermes repos (#23)
Merge main to all / rebase (push) Failing after 18s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 56s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 48s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 46s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 44s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 42s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 42s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m56s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 2m0s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m2s
Python package / build (pip list, 3.12.3) (push) Successful in 39s
Python package / build (pip list, 3.13) (push) Successful in 39s
Python package / build (pip list, 3.14) (push) Successful in 55s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 48s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 33s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 43s
## Summary

Migrates `backend/configs/torrserv_proxy.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/reverseproxy.git`) to the new `remotes=[{name, url}, ...]` form introduced by #3.

Source repo is `rudolf1/reverseproxy` (not `brencher`), so the three remotes point at:

- `github`       → `https://github.com/rudolf1/reverseproxy.git`
- `gitea_admin`  → `https://gitea.rudolf.keenetic.link/admin/reverseproxy`
- `gitea_hermes` → `https://gitea.rudolf.keenetic.link/hermes/reverseproxy`

Same pattern that already works in production for `brencher.py` and `brencher2.py`.

## Diff

```diff
-clone = GitClone(url="https://github.com/rudolf1/reverseproxy.git")
+clone = GitClone(remotes=[
+	{"name": "github",       "url": "https://github.com/rudolf1/reverseproxy.git"},
+	{"name": "gitea_admin",  "url": "https://gitea.rudolf.keenetic.link/admin/reverseproxy"},
+	{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/reverseproxy"},
+])
```

## Test plan

Manual test on prod: trigger brencher with this config, verify it clones from the gitea mirrors and `torrserv_proxy` stack deploys successfully.

Refs #22
Part of #18

---------

Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/23
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-30 22:29:00 +02:00
hermes 98ee950a1f fix(docker): cap swarm config name at 64 chars via sha256 fallback (#17)
Merge main to all / rebase (push) Failing after 14s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 39s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 46s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 39s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 39s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 38s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 41s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m50s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m57s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m5s
Python package / build (pip list, 3.12.3) (push) Successful in 34s
Python package / build (pip list, 3.13) (push) Successful in 34s
Python package / build (pip list, 3.14) (push) Successful in 35s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 31s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 31s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 32s
## What

Since #5 (Fix/show pipeline errors), brencher appends a version tail to every swarm config name so each deploy creates a new config object (swarm configs are immutable). For long base names plus the version string, the resulting prefixed name (stack_name_ + name) can exceed docker swarm's 64-char limit and `docker stack deploy` fails with `InvalidArgument`.

## Repro

```
[2026-08-29 19:38:46,063] failed to create config hermes_infra_brencher_state_proxy_conf_auto-16875f17-4cb8eeae-bfb3c220: Error response from daemon: rpc error: code = InvalidArgument desc = invalid name, only 64 [a-zA-Z0-9-_.] characters allowed
```

`brencher_state_proxy_conf_auto-16875f17-4cb8eeae-bfb3c220` is 57 chars, plus the 13-char `hermes_infra_` stack prefix = 70, over the 64-char limit.

## Fix

When the would-be full prefixed name is longer than 64 chars, replace the suffix with `sha256(full)[:32]`. 32 hex chars is comfortably unique per `(base_name, version)` pair and the prefixed name ends up at 12 + 1 + 32 = 45 chars, well under the limit.

Short names keep the readable `<name>_<version>` form (no change for the common case).

## Verified

Manual sanity check on the four configs currently in `hermes_squid`:

| base | version | compose name | docker name | length |
|---|---|---|---|---|
| squid_conf | auto-4cb8eeae-bfb3c220 | squid_conf_auto-4cb8eeae-bfb3c220 | hermes_infra_squid_conf_auto-4cb8eeae-bfb3c220 | 46 |
| brencher_state_proxy_conf | auto-16875f17-4cb8eeae-bfb3c220 | 7644a2983c7f6c7b09c6ed84bff0096d | hermes_infra_7644a2983c7f6c7b09c6ed84bff0096d | 45 |
| brencher_state_proxy_conf | auto-4cb8eeae-bfb3c220 | brencher_state_proxy_conf_auto-4cb8eeae-bfb3c220 | hermes_infra_brencher_state_proxy_conf_auto-4cb8eeae-bfb3c220 | 61 |
| nginx_hermes_conf | auto-16875f17-4cb8eeae-bfb3c220 | nginx_hermes_conf_auto-16875f17-4cb8eeae-bfb3c220 | hermes_infra_nginx_hermes_conf_auto-16875f17-4cb8eeae-bfb3c220 | 62 |

All under 64 chars; sha256 fallback only kicks in for the long case.

## Impact

- Fixes deploys of stacks that have a config with a long base name (e.g. `brencher_state_proxy_conf` in `hermes_infra`).
- Other stacks (`gitea`, `immich`, `torrserv_proxy`, etc.) are unaffected because their config names are short enough to fit the existing readable form.
- After merge + brencher self-deploy, the failed hermes_infra stack at `brencher-state-proxy` (failed since 2026-08-26) can finally be re-deployed and pick up the new `brencher-state-proxy.conf` with the resolver fix from #5's companion config rename.

## Related

- This unblocks `admin/ansible#21` (feat/stt-voice-faster-whisper) whose deploy was failing on the same length error.
- The companion nginx-side fix in `admin/ansible#21` (`brencher-state-proxy.conf`: `resolver` + `set $upstream`) will start working once this lands and the config object is actually recreated.Reviewed-on: http://192.168.1.96/admin/brencher/pulls/17
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-29 22:54:07 +02:00
admin e4f90b1f23 Fix/show pipeline errors (#5)
Merge main to all / rebase (push) Failing after 22s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 37s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 31s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 42s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 43s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m53s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m58s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 1m54s
Python package / build (pip list, 3.12.3) (push) Successful in 37s
Python package / build (pip list, 3.13) (push) Successful in 37s
Python package / build (pip list, 3.14) (push) Successful in 34s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 29s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 36s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 35s
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/5
2026-08-27 10:58:55 +02:00
hermesandhermes ba17865a49 feat(configs): point hermes_infra at gitea admin/hermes ansible repos (#8)
Merge main to all / rebase (push) Failing after 15s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 41s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 36s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 41s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 35s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 37s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m43s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m55s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 1m48s
Python package / build (pip list, 3.12.3) (push) Successful in 34s
Python package / build (pip list, 3.13) (push) Successful in 32s
Python package / build (pip list, 3.14) (push) Successful in 36s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 37s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 33s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 31s
## Summary
Switches `backend/configs/hermes_infra.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/hermes_infra.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why this config specifically
- `hermes_infra.py` is the config that **deploys `hermes_squid/stack-compose.yml`** — i.e. it is what brings the sidecars added in `admin/ansible#1` (the docker_socket_proxy + brencher-state-proxy from earlier today) to a running swarm. Pointing its clone at the unreachable github URL meant the deploy path was broken even though the file looked correct.
- `uber_backup.git` doesn't exist on gitea anyway (only `admin/ansible`, `hermes/ansible`, `admin/brencher`, `hermes/brencher` exist) — so the old URL was always wrong, not just unreachable.
- github.com returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (PR #4 / commit `653aee9`).

## Scope decision: only `hermes_infra.py` was migrated
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) still use the legacy form. They're intentionally left untouched — this PR keeps the migration scope narrow on `hermes_infra` only, as requested. A future follow-up PR can migrate them once the upstream uber_backup / reverseproxy mirrors exist on gitea (they don't right now).

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/hermes_infra.py` — OK
- Real import via `from configs import hermes_infra`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''` (not just unset)
  - `env.id == "hermes_infra"`, pipeline unchanged
  - `deploy step docker_compose_path == "hermes_squid/stack-compose.yml"`
  - `deploy step stack_name == "hermes_infra"`
- pytest: **26/26** passed on the no-docker / no-playwright suite (`test_dummy`, `test_caching_step`, `test_cached_env`, `test_git_integration`)
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`
  - no `github.com` in any remote URL

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible` (the live-deploy source). Sandbox can't simulate this.
- end-to-end `docker stack deploy hermes_infra` — needs a real swarm

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of the `hermes_infra` environment will clone from gitea instead of the unreachable github URL. Combined with `admin/ansible#1` (which adds the new services to `hermes_squid/stack-compose.yml`) and `admin/brencher#4` (which registers `docker_socket_proxy` in the PROFILES env var), the full chain is:
1. brencher clones ansible from `gitea.rudolf.keenetic.link/{hermes,admin}/ansible`
2. Checks out a merged branch containing the new sidecars
3. Builds (only `sarg` — `docker_socket_proxy` and `brencher-state-proxy` use upstream images, no build)
4. `docker stack deploy hermes_infra` rolls out the full stack

## Commit history
```
* 9a0cd90 (HEAD -> feat/point-hermes-infra-at-gitea, origin/feat/point-hermes-infra-at-gitea)
      feat(configs): point hermes_infra at gitea admin/hermes ansible repos
* e533174 Allow naming each GitClone remote explicitly (#3)         (upstream/main tip)
```

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/8
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-26 11:30:17 +02:00
hermes db3807e49a ci: speed up python-package.yml from ~13min to ~3-4min
Python package / build (3.12.3) (pull_request) Successful in 12m41s
Python package / build (3.13) (pull_request) Successful in 11m13s
Python package / build (3.14) (pull_request) Successful in 11m18s
The build job was running for ~13 minutes per PR even with tests
disabled (per PR #13). The actual cause: the matrix had
3 python-versions x 5 checkers = 15 separate CI jobs, each
re-installing the same requirements.txt + requirements-dev.txt
from scratch with no pip cache.

This commit:
  1. Enables pip caching via actions/setup-python@v5 cache: 'pip'
     keyed on the contents of requirements.txt and requirements-dev.txt.
     When those files don't change, pip install is near-instant.
     When they do change, only the diff downloads.

  2. Drops two debug-only matrix entries that were never real checks:
       python -c 'import sys; print(...)'  (debug)
       pip list                          (debug)
     They took up two of the five matrix cells but produced no
     actionable signal.

  3. Restructures the matrix so each Python version runs ONE job
     that does install + flake8 (2 invocations) + mypy sequentially,
     instead of one job per (version, checker) pair. Net: 3 jobs
     total instead of 15.

  4. Adds a concurrency: cancel-in-progress block so a new push
     to the same PR auto-cancels the previous in-flight run,
     saving runner minutes when iterating on a fix.

Verified:
  - YAML parses via PyYAML
  - jobs dict contains ['build'] only
  - matrix python versions: ['3.12.3', '3.13', '3.14']
  - total CI jobs: 3 (was 15)
  - actions/setup-python@v5 with cache: 'pip' + cache-dependency-path
    on requirements.txt and requirements-dev.txt

Expected wall-clock (rough estimate, depends on cache hits):
  Before: ~13 min (15 jobs, no pip cache, install on every job)
  After (cold): ~5-7 min (3 jobs, pip cache miss on first run)
  After (warm): ~3-4 min (3 jobs, pip cache hits)

If caching doesn't activate (e.g. self-hosted runner ignores cache:),
the win comes mostly from 15 -> 3 jobs (~5x faster).

Note: PR #14 (re-enable tests) is still open. If/when it merges, this
PR will need a follow-up to re-add the tests job with the same
cache setup applied.
2026-08-25 22:10:30 +00:00
hermesandhermes 70f4a4163e ci: comment out the tests job in python-package.yml (#13)
## Summary
Comments out the `tests:` job in `.github/workflows/python-package.yml`. The slow `pytest` matrix (with `playwright install --with-deps chromium`) no longer runs on every push or PR — only the cheap `build` job (lint + mypy) remains.

## Why
- `pytest` requires `playwright install --with-deps chromium` on every run, which dominates the CI time even though pytest is not currently the merge gate.
- The `build` job (flake8 + mypy) still runs and still catches real regressions cheaply.
- `workflow_dispatch` is unchanged — tests can still be triggered manually from the Actions UI when needed.

## What's in this PR
- `.github/workflows/python-package.yml` — single file, +35/-34
  - The `tests:` job (33 lines) is now commented with leading `# `
  - Everything else (triggers, `build:` job, matrix) is unchanged
  - YAML still parses cleanly; `jobs` dict now contains only `build`

## Verification
- `python -c "import yaml; yaml.safe_load(open('.github/workflows/python-package.yml'))"` — OK
- `wf['jobs']` contains `['build']` only (no `tests` key)
- `build.matrix.checker` still has all 5 checkers:
  - `python -c "import sys; ..."` (debug)
  - `pip list` (debug)
  - `flake8 . --select=E9,F63,F7,F82` (syntax errors)
  - `flake8 . --exit-zero --max-complexity=10` (complexity)
  - `mypy --no-incremental --config-file mypy.ini backend tests`
- triggers unchanged: `workflow_dispatch`, `push`, `pull_request` (all targeting `main`)

## How to re-enable
Remove the leading `# ` from each commented line, or delete the commented block entirely and rely on git history.

## What was NOT verified
- The actual GitHub Actions runner behaviour change — this repo runs on Gitea Actions (the project's primary CI is the brencher pipeline itself, not Gitea-hosted runners for these workflows). Gitea's Actions runner is GitHub Actions–compatible and will see the `tests` job as absent and skip it.

## Commit history
```
* e271b2e (HEAD -> feat/disable-ci-tests, origin/feat/disable-ci-tests)
      ci: comment out the tests job in python-package.yml
* e533174 Allow naming each GitClone remote explicitly (#3)         (upstream/main tip)
```

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/13
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-25 23:24:49 +02:00
e5331745be Allow naming each GitClone remote explicitly (#3)
Merge main to all / rebase (push) Failing after 19s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 38s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 38s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 52s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 41s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 40s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m55s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 2m1s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m1s
Python package / build (pip list, 3.12.3) (push) Successful in 38s
Python package / build (pip list, 3.13) (push) Successful in 40s
Python package / build (pip list, 3.14) (push) Successful in 41s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 35s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 37s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 40s
Python package / tests (pytest, 3.12.3) (push) Failing after 7m43s
Python package / tests (pytest, 3.13) (push) Failing after 7m7s
Python package / tests (pytest, 3.14) (push) Failing after 7m36s
Implements the review comment on #2:

> I would like to give names for remotes

## Summary

Adds a `remotes=[{"name": ..., "url": ...}, ...]` constructor to `GitClone` so the config author controls how each URL maps to a git remote. The existing `url=str` / `url=[str]` shorthand is kept as backward-compatible (positional list still maps to `origin`, `repo1`, `repo2`, ...), so PR #2's `test_git_clone_multiple_urls` continues to pass.

## Changes

- **`backend/steps/git.py`** — `GitClone.__init__` now accepts either `url=` or `remotes=`. With `remotes=`, every entry must carry a non-empty unique `name` and `url`; duplicates and missing fields raise `ValueError`. The constructor rejects passing both or neither. `self.remotes` becomes the source of truth; `self.urls` is a derived list (kept for any introspection). `CheckoutMerged.find_desired_commits` falls back to the configured primary remote instead of the hardcoded `'origin'` when a branch key has no remote prefix, so configs that don't name a remote `origin` still work.
- **`backend/configs/brencher.py`, `brencher2.py`** — switch from positional list to named remotes `github`, `gitea_admin`, `gitea_hermes`.
- **`tests/test_git_integration.py`** — adds `test_git_clone_named_remotes` (asserts the user-chosen names appear verbatim and `origin`/`repo1` do not leak) and `test_git_clone_remotes_validation` (covers every rejection path: empty list, missing/empty name, missing/empty url, duplicate name, both-kwargs, no-kwargs).

## Diff vs #2

```
backend/configs/brencher.py    |  8 ++---
backend/configs/brencher2.py   |  8 ++---
backend/steps/git.py           | 76 +++++++++++++++++++++++++++++++++++++++-------
tests/test_git_integration.py  | 75 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 147 insertions(+), 20 deletions(-)
```

Targets `admin/multiple_remotes` so this lands on top of #2. After #2 merges, this is a clean follow-up; if #2 is reworked, this branch will need a rebase.

## Verification

- `python3 -m py_compile` on all 4 changed files: clean.
- AST-driven call-shape check: every `GitClone(...)` in both configs uses the new `remotes=` kwarg with exactly `{github, gitea_admin, gitea_hermes}`; no positional-list leftover in configs.
- Executed the new `__init__` in isolation against a stubbed `super()` and exercised 12 cases: 3 OK shapes (url=str, url=[..], remotes=[valid]) and 9 rejection shapes (empty list, missing/empty name, missing/empty url, duplicate name, both-kwargs, no-kwargs, non-dict entry). All 12 produced the asserted `ValueError`/`TypeError` with the expected message substring.
- Could not run `pytest` in this sandbox (PyPI is blocked by the proxy and no `pytest`/`gitpython` is pre-installed). The two new tests and PR #2's existing test should be run in a real env before merge.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Co-authored-by: rudolf <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/3
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-25 18:53:12 +02:00
admin 94a858d8ef Version on UI + gitea 2026-08-12 21:09:07 +02:00
admin 57393c933c hermes (#83)
* hermes
2026-08-05 21:26:07 +02:00
adminandGitHub Actions 45898cc4e2 x (#80)
* Squash

* Comment out branch state conflict error handling

Comment out error broadcasting logic for branch state conflicts.

---------

Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2026-08-04 14:29:26 +02:00
admin 2a14d70fab playwright tests. refactor backend (#79)
* refactor backend
* Playwright test
2026-05-10 20:14:50 +02:00
31 changed files with 1794 additions and 588 deletions
@@ -0,0 +1,53 @@
---
name: playwright-brencher
description: "Write or refactor Playwright tests for brencher UI. Use when creating page helpers, extracting browser code from tests, or enforcing set_/verify_ method naming."
---
# Playwright Brencher Conventions
Use this skill when writing or refactoring Playwright tests in this repository.
## Required API Shape
Create and use a context manager in this style:
```python
with brencher_page(APP_URL) as page:
page.verify_dry_run_on()
page.set_dry_run_off()
page.verify_dry_run_off()
```
## Naming Rules
1. Action methods that change UI/application state must use `set_...` names.
2. Verification methods (assertions, waits, checks) must use `verify_...` names.
3. Keep low-level selector helpers private (prefix `_`).
4. Prefer one page-object class per screen/area (`BrencherPage` for the current app page).
## Structure Rules
1. Put shared Playwright helpers in `tests/playwright_helper.py`.
2. Keep tests focused on scenario flow; move DOM selectors and waiting logic into page helper methods.
3. Use context manager functions to own Playwright/browser lifecycle.
4. Inline trivial one-use logic where it improves readability and does not duplicate behavior.
## Method Guidance
Recommended style for page object methods:
- `set_*`: click/toggle/type actions.
- `verify_*`: assert visible state, button title/text, or wait until target state is reached.
Example method names:
- `set_dry_run_off`
- `verify_dry_run_on`
- `verify_dry_run_off`
## Review Checklist
1. Test uses `with brencher_page(...) as page` style.
2. No direct `sync_playwright()` block remains in the test body when helper abstraction is intended.
3. Action and verification methods follow `set_` / `verify_` naming.
4. Selector details are encapsulated in private helper methods.
+21 -25
View File
@@ -1,6 +1,3 @@
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
name: Python package
on:
@@ -10,41 +7,40 @@ on:
pull_request:
branches: [ "main" ]
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: [
# "3.11",
"3.12.3",
"3.13",
"3.14",
# "3.15",
]
checker: [
"python -c \"import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')\"",
"pip list",
"flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics",
"flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics",
# "ty check backend",
"mypy --no-incremental --config-file mypy.ini backend tests",
"pytest",
]
python-version: ["3.12.3", "3.13", "3.14"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}, ${{ matrix.checker }}
uses: actions/setup-python@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: |
requirements.txt
requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install flake8 pytest ty
pip install -r requirements.txt
pip install -r requirements-dev.txt
- name: Run checker
run: ${{ matrix.checker }}
- name: Lint (flake8)
run: |
flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Type-check (mypy)
run: mypy --no-incremental --config-file mypy.ini backend tests
+31
View File
@@ -0,0 +1,31 @@
FROM nginx:1.27-alpine
COPY .git /tmp/repo/.git
# Configure nginx to listen on 5001 to match container port expectations.
RUN sed -i 's/listen 80;/listen 5001;/' /etc/nginx/conf.d/default.conf
# Build a custom landing page directly in the image and inject git revision.
RUN set -eux; \
apk add --no-cache --virtual .git-deps git; \
GIT_REVISION="$(git --git-dir=/tmp/repo/.git rev-parse --short HEAD 2>/dev/null || echo unknown)"; \
{ \
echo '<!doctype html>'; \
echo '<html lang="en">'; \
echo '<head>'; \
echo ' <meta charset="utf-8">'; \
echo ' <meta name="viewport" content="width=device-width, initial-scale=1">'; \
echo ' <title>Brencher Test App</title>'; \
echo '</head>'; \
echo '<body>'; \
echo ' <h1>Nginx Is Running</h1>'; \
echo ' <p>Custom landing page for the Brencher test app container.</p>'; \
echo ' <p>Port: 5001</p>'; \
echo " <p>Git revision: ${GIT_REVISION}</p>"; \
echo '</body>'; \
echo '</html>'; \
} > /usr/share/nginx/html/index.html; \
rm -rf /tmp/repo/.git; \
apk del .git-deps
EXPOSE 5001
+89 -388
View File
@@ -1,22 +1,16 @@
import asyncio
import json
import logging
import os
import signal
import sys
import threading
import traceback
from dataclasses import asdict, replace
from typing import Any, Dict, List, Optional, TypeVar, Tuple
from typing import Any, Callable, Dict, List, Optional
from discover_envs import build_environments, parse_arguments
from utils import sigchld_handler
from dotenv import load_dotenv
from fastapi import FastAPI, Response, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse
from enironment import Environment, wrap_in_cached, SharedStateHolder, SharedStateConflictError, SharedState, get_step
from processing import reset_caches
from secondary import SecondaryManager
from enironment import Environment, wrap_in_cached, SharedStateHolder, get_step
from steps.git import GitClone
from steps.step import CachingStep
@@ -29,403 +23,100 @@ load_dotenv("local.env")
load_dotenv('/run/secrets/brencher-secrets')
class DataclassJSONEncoder(json.JSONEncoder):
def default(self, o: Any) -> Any:
if hasattr(o, '__dataclass_fields__'):
return asdict(o)
if isinstance(o, BaseException):
return str(o)
try:
return super().default(o)
except TypeError:
return str(o)
def custom_json_dumps(obj: Any) -> str:
return json.dumps(obj, cls=DataclassJSONEncoder)
app = FastAPI()
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
FRONTEND_DIR = os.path.join(os.path.dirname(__file__), '../frontend')
# In-memory state
environments: Dict[str, Environment] = {}
state_lock = threading.Lock()
# Active WebSocket connections
ws_connections: Dict[WebSocket, Dict[str, Any]] = {}
# Event loop reference for thread-safe async calls
_event_loop: Optional[asyncio.AbstractEventLoop] = None
environment_update_event = threading.Event()
# --- Static File Serving ---
@app.get("/")
async def serve_index() -> FileResponse:
return FileResponse(os.path.join(FRONTEND_DIR, 'index.html'))
@app.get("/state")
async def serve_state() -> Response:
return Response(content=custom_json_dumps(get_global_envs_to_emit()), media_type="application/json")
@app.get("/branches")
async def serve_branches_route() -> Response:
return Response(content=custom_json_dumps(get_global_branches_to_emit()), media_type="application/json")
@app.get("/{path:path}")
async def serve_static(path: str) -> FileResponse:
file_path = os.path.join(FRONTEND_DIR, path)
if os.path.exists(file_path) and os.path.isfile(file_path):
return FileResponse(file_path)
return FileResponse(os.path.join(FRONTEND_DIR, 'index.html'))
# --- Utility Functions ---
T = TypeVar('T')
def merge_dicts(a: Dict[str, T], b: Dict[str, T]) -> Dict[str, T]:
result: Dict[str, T] = {}
for k in (a.keys() | b.keys()):
if k not in a or k not in b:
result[k] = a.get(k, b.get(k)) # type: ignore[assignment]
continue
if (
isinstance(a[k], dict)
and isinstance(b[k], dict)
):
result[k] = merge_dicts(a[k], b[k]) # type: ignore[arg-type,assignment]
else:
result[k] = b[k]
return result
def get_local_envs_to_emit() -> Dict[str, Dict[str, Any]]:
env_dtos: Dict[str, Dict[str, Any]] = {}
for env in environments.values():
pipeline_state: List[Dict[str, Any]] = []
for r in env.pipeline:
try:
if isinstance(r, CachingStep):
result = r._result
else:
result = r.progress()
if isinstance(result, BaseException):
stack = traceback.format_exception(type(result), result, result.__traceback__)
pipeline_state.append({
"name": r.name,
"status": [str(result), stack],
"error": True,
"is_running": True,
})
else:
pipeline_state.append({
"name": r.name,
"status": result,
"is_running": False,
})
except BaseException as e:
stack = traceback.format_exception(type(e), e, e.__traceback__)
pipeline_state.append({
"name": r.name,
"status": [str(e), stack],
"error": True,
"is_running": True,
})
env_dtos[env.id] = {'id': env.id }
env_dtos[env.id]['pipeline'] = pipeline_state
try:
shared_state = env.state.progress()
env_dtos[env.id]['branches'] = shared_state.branches
env_dtos[env.id]['branches_token'] = shared_state.token
env_dtos[env.id]['dry'] = shared_state.dry
except BaseException:
pass
return env_dtos
def get_global_envs_to_emit() -> Any:
global environments
local_envs = get_local_envs_to_emit()
merged: Dict[str, Any] = local_envs
for connector in secondaryManager or []:
v = connector.environments
common_keys = set(merged.keys()) & set(v.keys())
if len(common_keys) > 0:
_schedule_async(broadcast_error({'message': f"Conflict: both master and secondary have environment with id {common_keys}"}))
merged = merge_dicts(merged, v)
return merged
def get_local_branches_to_emit() -> Dict[str, Dict[str, List[Any]]]:
global environments
branches: Dict[str, Dict[str, List[Any]]] = {}
for k, env in environments.items():
branches[k] = {}
try:
step = get_step(env.pipeline, GitClone)
branches[k] = {**step.get_branches()}
except BaseException as e:
stack = traceback.format_exception(type(e), e, e.__traceback__)
logger.error(f"Error fetching branches for environment {env.id}: {str(e)}\n{''.join(stack)}")
return branches
def get_global_branches_to_emit() -> Dict[str, Dict[str, List[Any]]]:
local_branches: Dict[str, Dict[str, List[Any]]] = get_local_branches_to_emit()
merged: Dict[str, Dict[str, List[Any]]] = local_branches
for connector in secondaryManager or []:
merged = merge_dicts(merged, connector.branches)
return merged
# --- Async Helper for Calling from Sync Threads ---
def _schedule_async(coro: Any) -> None:
"""Schedule a coroutine from a sync thread using the saved event loop."""
if _event_loop is not None and _event_loop.is_running():
asyncio.run_coroutine_threadsafe(coro, _event_loop)
# --- WebSocket Broadcasting Functions ---
async def broadcast(event: str, data: Any) -> None:
"""Broadcast a message to all connected WebSocket clients that have not seen this data yet."""
disconnected = set()
message = custom_json_dumps({event: data})
for websocket in ws_connections.keys():
if ws_connections[websocket].get(event) == message:
continue
try:
await websocket.send_text(message)
ws_connections[websocket][event] = message
except Exception as e:
logger.error(f"Error sending to websocket: {e}")
disconnected.add(websocket)
for ws in disconnected:
ws_connections.pop(ws, None)
async def broadcast_all() -> None:
await broadcast("branches", get_global_branches_to_emit())
await broadcast("environments", get_global_envs_to_emit())
async def broadcast_branches(data: Any) -> None:
await broadcast("branches", data)
async def broadcast_environments(data: Any) -> None:
await broadcast("environments", data)
async def broadcast_error(data: Any) -> None:
await broadcast("error", data)
# --- Single WebSocket Endpoint ---
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket) -> None:
await websocket.accept()
ws_connections[websocket] = {}
try:
# Send initial state on connect and record it so duplicate broadcasts are suppressed
branches_payload = custom_json_dumps({"branches": get_global_branches_to_emit()})
envs_payload = custom_json_dumps({"environments": get_global_envs_to_emit()})
await websocket.send_text(branches_payload)
ws_connections[websocket]["branches"] = branches_payload
await websocket.send_text(envs_payload)
ws_connections[websocket]["environments"] = envs_payload
while True:
data = await websocket.receive_text()
message = json.loads(data)
if "update" in message:
update_data = message.get("update") or {}
logger.info(f"Received environment update: {update_data}")
if secondaryManager:
await secondaryManager.send({"update": update_data})
id = update_data.get('id', '')
if id == '':
reset_caches(list(environments.values()))
elif id not in environments.keys() and id not in { j for it in secondaryManager or [] for j in it.environments.keys()}:
logger.warning(f"Received update for unknown environment id {id}")
continue
elif id in environments.keys():
env = environments.get(id, None)
expected_token = update_data.get('token', '')
if 'branches' in update_data:
if not env:
raise RuntimeError(f"Unknown env {update_data.get('id', '')}")
env.state.set_branches(update_data.get('branches', []), expected_token=expected_token)
if 'dry' in update_data:
if not env:
raise RuntimeError(f"Unknown env {update_data.get('id', '')}")
env.state.set_dry(bool(update_data['dry']), expected_token)
if env:
p = get_step(env.pipeline, type(env.state))
if isinstance(p, CachingStep):
p.reset()
logger.info(f"Updated environment {env.id} branches to {update_data.get('branches')}, dry={update_data.get('dry')}")
environment_update_event.set()
await broadcast_environments(get_global_envs_to_emit())
except WebSocketDisconnect as e:
ws_connections.pop(websocket, None)
except Exception as e:
logger.error(f"WebSocket error: {e}:{traceback.format_exc()}")
await broadcast_error({'message': f'{e}:{traceback.format_exc()}'})
# await websocket.close()
# ws_connections.pop(websocket, None)
# --- SIGCHLD Handler ---
def sigchld_handler(signum: int, frame: Any) -> None:
"""Reap zombie processes."""
while True:
try:
pid, status = os.waitpid(-1, os.WNOHANG)
if pid == 0:
break
except ChildProcessError:
break
# --- Startup Event ---
secondaryManager: Optional[SecondaryManager] = None
@app.on_event("startup")
async def startup_event() -> None:
global _event_loop, secondaryManager
_event_loop = asyncio.get_event_loop()
secondaryManager = SecondaryManager(
urls = os.getenv("SECONDARY_BRENCHER", ""),
on_update=lambda: broadcast_all(),
)
# --- App Class (used by CLI and integration tests) ---
class App:
def __init__(self, cli_env_ids_str: str, dry_run: bool = False) -> None:
global environments
signal.signal(signal.SIGCHLD, sigchld_handler)
def __init__(self, environments: Dict[str, Environment]) -> None:
self.environments: Dict[str, Environment] = {id: wrap_in_cached(e) for id, e in environments.items()}
self.state_lock = threading.Lock()
self.environment_update_event = threading.Event()
self.emit_callback: Optional[Callable[[], None]] = None
import configs.brencher
import configs.brencher2
import configs.brencher_local2
import configs.brencher_local1
import configs.torrserv_proxy
import configs.immich
import configs.registry
import configs.gmail_mcp
environments_l: List[Environment] = [
configs.brencher.brencher,
configs.brencher2.brencher2,
configs.brencher_local2.brencher_local2,
configs.brencher_local1.brencher_local1,
configs.torrserv_proxy.torrserv_proxy,
configs.immich.immich,
configs.registry.registry,
configs.gmail_mcp.gmail_mcp,
]
environments = {e.id: e for e in environments_l}
logger.info(f"cli_env_ids {cli_env_ids_str}")
if len(cli_env_ids_str) > 0 and cli_env_ids_str[0] == '-':
cli_env_ids = cli_env_ids_str[1:].split(',')
cli_env_ids = [x for x in cli_env_ids if len(x) > 0]
logger.info(f"cli_env_ids (minus) {cli_env_ids}")
if cli_env_ids and len(cli_env_ids) > 0:
environments = {k: e for k, e in environments.items() if k not in cli_env_ids}
else:
cli_env_pairs = {x[0]: x[1] if len(x) > 1 else None for x in [x.split(":")
for x in cli_env_ids_str.split(',')
if len(x) > 0
]
}
logger.info(f"cli_env_ids {cli_env_pairs}")
if cli_env_pairs and len(cli_env_pairs) > 0:
environments = {k: e for k, e in environments.items() if k in cli_env_pairs.keys()}
for k, v in cli_env_pairs.items():
if v is not None:
if k in environments:
env = environments[k]
for step in env.pipeline:
resolve_step = step
if isinstance(step, CachingStep):
resolve_step = step._step
if isinstance(resolve_step, SharedStateHolder):
resolve_step.set_branches([(v, 'HEAD')])
logger.info(f"Overriding environment {k} branches to {(v, 'HEAD')}")
else:
logger.warning(f"Environment {k} not found to override branches")
def get_local_envs_to_emit(self) -> Dict[str, Dict[str, Any]]:
env_dtos: Dict[str, Dict[str, Any]] = {}
for env in self.environments.values():
pipeline_state: List[Dict[str, Any]] = []
for r in env.pipeline:
try:
if isinstance(r, CachingStep):
result = r._result
else:
logger.info(f"No branch override for environment {k}")
if dry_run:
for id, e in environments.items():
e.state.set_dry(True)
result = r.progress()
logger.info(f"Resulting profiles {environments.keys()}")
if isinstance(result, BaseException):
stack = traceback.format_exception(type(result), result, result.__traceback__)
pipeline_state.append({
"name": r.name,
"status": [str(result), stack],
"error": True,
"is_running": False,
})
else:
pipeline_state.append({
"name": r.name,
"status": result,
"is_running": False,
})
except BaseException as e:
stack = traceback.format_exception(type(e), e, e.__traceback__)
pipeline_state.append({
"name": r.name,
"status": [str(e), stack],
"error": True,
"is_running": False,
})
env_dtos[env.id] = {'id': env.id }
env_dtos[env.id]['pipeline'] = pipeline_state
try:
shared_state = env.state.progress()
env_dtos[env.id]['branches'] = shared_state.branches
env_dtos[env.id]['branches_token'] = shared_state.token
env_dtos[env.id]['dry'] = shared_state.dry
except BaseException:
pass
return env_dtos
environments = {id: wrap_in_cached(e) for id, e in environments.items()}
def get_local_branches_to_emit(self) -> Dict[str, Dict[str, List[Any]]]:
branches: Dict[str, Dict[str, List[Any]]] = {}
for k, env in self.environments.items():
branches[k] = {}
try:
step = get_step(env.pipeline, GitClone)
branches[k] = {**step.get_branches()}
except BaseException as e:
stack = traceback.format_exception(type(e), e, e.__traceback__)
logger.error(f"Error fetching branches for environment {env.id}: {str(e)}\n{''.join(stack)}")
return branches
def processing_thread(self) -> None:
while True:
import processing
def emit_envs() -> None:
_schedule_async(broadcast_branches(get_global_branches_to_emit()))
_schedule_async(broadcast_environments(get_global_envs_to_emit()))
emit = self.emit_callback or (lambda: None)
with state_lock:
with self.state_lock:
logger.info(f"Processing")
if processing.process_all_jobs(list(environments.values()), lambda: emit_envs()):
environment_update_event.wait(timeout=1 * 5)
if processing.process_all_jobs(list(self.environments.values()), emit):
self.environment_update_event.wait(timeout=1 * 5)
else:
environment_update_event.wait(timeout=1 * 60)
environment_update_event.clear()
self.environment_update_event.wait(timeout=1 * 60)
self.environment_update_event.clear()
def run(self) -> None:
processing = threading.Thread(target=self.processing_thread)
processing.daemon = True
processing.start()
def runWeb(self, port: int) -> None:
self.run()
import uvicorn
uvicorn.run(app, host='0.0.0.0', port=port, log_level='info')
def runHeadless(self) -> None:
"""Run the processing loop on the current thread; blocks forever."""
self.processing_thread()
def runWeb(self, port: int) -> None:
import web
web_app = web.WebApp(core=self, port=port)
self.emit_callback = web_app.emit_envs
self.run()
web_app.start()
if __name__ == '__main__':
cli_env_ids_list = sys.argv[1:]
@@ -435,5 +126,15 @@ if __name__ == '__main__':
else:
cli_env_ids_str = cli_env_ids_list[0]
app1 = App(cli_env_ids_str, 'dry' in sys.argv[1:])
app1.runWeb(port=(5007 if 'noweb' in sys.argv[1:] else 5001))
signal.signal(signal.SIGCHLD, sigchld_handler)
environments = build_environments(cli_env_ids_str)
if 'dry' in sys.argv[1:]:
for e in environments.values():
e.state.set_dry(True)
app1 = App(environments)
if 'noweb' in sys.argv[1:]:
app1.runHeadless()
else:
app1.runWeb(5001)
+10 -6
View File
@@ -5,7 +5,11 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/brencher.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/brencher"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/brencher"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher",
@@ -45,10 +49,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="docker-compose.yml",
)
checkPing = UrlCheck(
url="https://brencher.rudolf.keenetic.link/state",
expected=checkPingF,
)
#checkPing = UrlCheck(
# url="https://brencher.rudolf.keenetic.link/state",
# expected=checkPingF,
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://brencher.rudolf.keenetic.link/",
@@ -68,7 +72,7 @@ brencher = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
]
)
+10 -6
View File
@@ -21,7 +21,11 @@ def checkPingF(obj: Any) -> None:
raise Exception(f"brencher2 check failed for: {v}")
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/brencher.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/brencher"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/brencher"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher2",
@@ -72,10 +76,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="docker-compose.yml",
)
checkPing = UrlCheck(
url="https://brencher.rudolf.keenetic.link/state",
expected=checkPingF,
)
#checkPing = UrlCheck(
# url="https://brencher.rudolf.keenetic.link/state",
# expected=checkPingF,
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://brencher.rudolf.keenetic.link/",
@@ -98,7 +102,7 @@ brencher2 = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
],
)
+59
View File
@@ -0,0 +1,59 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="gitea",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=None,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="gitea",
docker_compose_path="poc/gitea/stack-compose.yml",
)
# checkPing1 = UrlCheck(
# url="http://100.70.193.97:8087/api/status",
# expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
# )
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://gitea.rudolf.keenetic.link",
}
})
__all__ = ["gitea"]
gitea = Environment(
id="gitea",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
# checkPing1,
logUrls
]
)
+59
View File
@@ -0,0 +1,59 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="hermes",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=None,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="hermes",
docker_compose_path="poc/hermes/stack-compose.yml",
)
checkPing1 = UrlCheck(
url="http://100.70.193.97:8087/api/status",
expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://hermes.rudolf.keenetic.link",
}
})
__all__ = ["hermes"]
hermes = Environment(
id="hermes",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing1,
logUrls
]
)
+94
View File
@@ -0,0 +1,94 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="hermes_infra",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=None)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
buildDocker = DockerComposeBuild(clone,
docker_repo_username="",
docker_repo_password="",
docker_compose_path="hermes_squid/stack-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
build_cache = True,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
# "user_group": "1000:137"
},
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=buildDocker,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="hermes_infra",
docker_compose_path="hermes_squid/stack-compose.yml",
)
# checkPing1 = UrlCheck(
# url="http://100.70.193.97:8088/api/status",
# expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
# )
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://hermes.rudolf.keenetic.link",
}
})
__all__ = ["hermes_infra"]
hermes_infra = Environment(
id="hermes_infra",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
# checkPing1,
logUrls
]
)
+5 -5
View File
@@ -31,10 +31,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="poc/immich/stack-compose.yml",
)
checkPing = UrlCheck(
url="https://immich.rudolf.keenetic.link/api/server/ping",
expected={"res": "pong"},
)
#checkPing = UrlCheck(
# url="https://immich.rudolf.keenetic.link/api/server/ping",
# expected={"res": "pong"},
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://immich.rudolf.keenetic.link",
@@ -52,7 +52,7 @@ immich = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
]
)
+5 -1
View File
@@ -3,7 +3,11 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/reverseproxy.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/reverseproxy.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/reverseproxy"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/reverseproxy"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="torrserv_proxy",
+71
View File
@@ -0,0 +1,71 @@
import importlib
import logging
import pkgutil
from dataclasses import dataclass, field
from typing import Dict, List, Optional
from enironment import Environment, SharedStateHolder
from steps.step import CachingStep
logger = logging.getLogger(__name__)
@dataclass
class ParsedEnvArgs:
excludes: List[str] = field(default_factory=list)
includes: Dict[str, Optional[str]] = field(default_factory=dict)
def parse_arguments(cli_env_ids_str: str) -> ParsedEnvArgs:
logger.info(f"cli_env_ids {cli_env_ids_str}")
if len(cli_env_ids_str) > 0 and cli_env_ids_str[0] == '-':
excludes = [x for x in cli_env_ids_str[1:].split(',') if len(x) > 0]
logger.info(f"cli_env_ids (minus) {excludes}")
return ParsedEnvArgs(excludes=excludes)
includes: Dict[str, Optional[str]] = {
x[0]: x[1] if len(x) > 1 else None
for x in [x.split(":") for x in cli_env_ids_str.split(',') if len(x) > 0]
}
logger.info(f"cli_env_ids {includes}")
return ParsedEnvArgs(includes=includes)
def _discover_environments() -> Dict[str, Environment]:
import configs
found: Dict[str, Environment] = {}
seen: set[int] = set()
for module_info in pkgutil.iter_modules(configs.__path__, prefix=f"{configs.__name__}."):
module = importlib.import_module(module_info.name)
for value in vars(module).values():
if isinstance(value, Environment) and id(value) not in seen:
seen.add(id(value))
found[value.id] = value
return found
def build_environments(cli_env_ids_str: str) -> Dict[str, Environment]:
environments: Dict[str, Environment] = _discover_environments()
args: ParsedEnvArgs = parse_arguments(cli_env_ids_str)
if args.excludes:
environments = {k: e for k, e in environments.items() if k not in args.excludes}
elif args.includes:
environments = {k: e for k, e in environments.items() if k in args.includes}
for k, v in args.includes.items():
if v is None:
logger.info(f"No branch override for environment {k}")
continue
if k not in environments:
logger.warning(f"Environment {k} not found to override branches")
continue
env = environments[k]
for step in env.pipeline:
resolve_step = step
if isinstance(step, CachingStep):
resolve_step = step._step
if isinstance(resolve_step, SharedStateHolder):
resolve_step.set_branches([(v, 'HEAD')])
logger.info(f"Overriding environment {k} branches to {(v, 'HEAD')}")
logger.info(f"Resulting profiles {environments.keys()}")
return environments
+69 -5
View File
@@ -1,3 +1,4 @@
import hashlib
import logging
import os
import re
@@ -197,6 +198,32 @@ class DockerSwarmDeploy(AbstractStep[str]):
content = f.read()
content = re.sub(r'\$\{([^}]+)\}', lambda m: env.get(m.group(1), ""), content)
compose = yaml.safe_load(content)
version = env["version"]
if "configs" in compose:
sanitized_version = re.sub(r'[^a-zA-Z0-9_.-]', '_', version)
full_prefix = f"{self.stack_name}_"
def _config_full_name(base_name: str) -> str:
full = f"{full_prefix}{base_name}_{sanitized_version}"
if len(full) <= 64:
return f"{base_name}_{sanitized_version}"
return hashlib.sha256(full.encode("utf-8")).hexdigest()[:32]
config_names = {
name: _config_full_name(name)
for name in compose["configs"]
}
compose["configs"] = {
config_names.get(name, name): config
for name, config in compose["configs"].items()
}
for service in compose.get("services", {}).values():
for config in service.get("configs", []):
if isinstance(config, str):
config_index = service["configs"].index(config)
service["configs"][config_index] = config_names.get(config, config)
elif isinstance(config, dict) and "source" in config:
config["source"] = config_names.get(config["source"], config["source"])
if "services" in compose:
for svc in compose["services"].values():
if "build" in svc:
@@ -268,6 +295,7 @@ class DockerSwarmDeploy(AbstractStep[str]):
# Deploy stack to Docker Swarm
cmd = [
"docker", "stack", "deploy",
"--detach=false",
"--prune",
"-c", tmp_compose_path,
self.stack_name
@@ -279,12 +307,27 @@ class DockerSwarmDeploy(AbstractStep[str]):
swarmEnv = {k: v for k, v in dotenv_values(os.path.join(os.path.dirname(tmp_compose_path), ".env")).items()
if v is not None}
# merge_dicts(swarmEnv, env)
result = subprocess.run(cmd, capture_output=True, text=True)
# , cwd=os.path.dirname(tmp_compose_path), env=swarmEnv)
result = subprocess.run(cmd, capture_output=True, text=True, cwd=os.path.dirname(tmp_compose_path))
#, env=swarmEnv)
os.remove(tmp_compose_path)
if result.returncode != 0:
logger.error(f"Stack deploy failed: {result.stderr}")
raise RuntimeError(f"Stack deploy failed: {result.stderr}")
logger.info(f"docker stack deploy stdout: {result.stdout or '<empty>'}")
logger.info(f"docker stack deploy stderr: {result.stderr or '<empty>'}")
stderr_lower = result.stderr.lower()
docker_error_markers = (
"failed to ",
"error response from daemon",
"rpc error",
"denied",
"no such image",
)
if result.returncode != 0 or any(marker in stderr_lower for marker in docker_error_markers):
deploy_output = (
f"exit code: {result.returncode}\n"
f"stdout:\n{result.stdout or '<empty>'}\n"
f"stderr:\n{result.stderr or '<empty>'}"
)
logger.error(f"Stack deploy failed:\n{deploy_output}")
raise RuntimeError(f"Stack deploy failed:\n{deploy_output}")
logger.info(f"Stack deployed successfully: {result.stdout}")
# Check health of all deployed services
@@ -308,4 +351,25 @@ class DockerSwarmDeploy(AbstractStep[str]):
if not_running:
raise BaseException(f"Services not started yet: {', '.join(not_running)}")
version_mismatches = []
for svc_name, svc in expected_services.items():
full_svc_name = f"{self.stack_name}_{svc_name}"
services_list = client.services.list(filters={"name": full_svc_name})
if not services_list:
continue
service_labels = services_list[0].attrs.get("Spec", {}).get("TaskTemplate", {}).get("ContainerSpec", {}).get("Labels", {})
expected_version = svc.get("labels", {}).get("org.brencher.version")
actual_version = service_labels.get("org.brencher.version")
if expected_version != actual_version:
version_mismatches.append(
f"{full_svc_name} expected version {expected_version}, got {actual_version}"
)
update_status = services_list[0].attrs.get("UpdateStatus", {})
if update_status.get("State") in {"paused", "rollback_paused", "rollback_started"}:
version_mismatches.append(
f"{full_svc_name} Swarm update {update_status.get('State')}: {update_status.get('Message', '')}"
)
if version_mismatches:
raise RuntimeError(f"Deployment did not apply requested versions: {'; '.join(version_mismatches)}")
return f"Stack deployed successfully: {result.stdout}"
+4
View File
@@ -71,6 +71,10 @@ class DockerImageBuild(AbstractStep[DockerImageBuildResult]):
if len(existing_image) > 0 and not self.nocache:
logger.info(f"Image {full_image} already exists locally with ID: {existing_image}")
build_logs: List[str] = []
elif self.env.dry:
logger.info(f"DRY RUN: Would build image {full_image}")
raise BaseException(f"Dry run - skipping actual build for {full_image}")
else:
# Build the image
build_logs = []
+118 -47
View File
@@ -16,11 +16,57 @@ logger = logging.getLogger(__name__)
class GitClone(AbstractStep[str]):
def __init__(self, url: str, repo_path: str | None = None,
branchNamePrefix: str = "", credEnvPrefix: str = "GIT",
def __init__(self,
url: str | List[str] | None = None,
remotes: List[Dict[str, str]] | None = None,
repo_path: str | None = None,
branchNamePrefix: str = "",
credEnvPrefix: str = "GIT",
**kwargs: Any):
super().__init__(**kwargs)
self.url = url
# `remotes` is the explicit form: a list of {"name": "...", "url": "..."}.
# `url` is kept as a shorthand: a single string becomes one remote named
# "origin"; a list of strings becomes remotes named "origin", "repo1",
# "repo2", ... (matches the behavior introduced by the "multiple remotes"
# change so existing positional-list callers keep working).
if remotes is not None and url is not None:
raise ValueError("Pass either 'url' or 'remotes', not both")
normalized: List[Dict[str, str]] = []
if remotes is not None:
seen: Set[str] = set()
for entry in remotes:
if not isinstance(entry, dict):
raise TypeError(
f"Each entry in 'remotes' must be a dict with 'name' and 'url', got {type(entry).__name__}"
)
name = entry.get("name")
url_value = entry.get("url")
if not name or not isinstance(name, str):
raise ValueError("Each remote requires a non-empty 'name' (string)")
if not url_value or not isinstance(url_value, str):
raise ValueError(f"Remote {name!r} requires a non-empty 'url' (string)")
if name in seen:
raise ValueError(f"Duplicate remote name: {name!r}")
seen.add(name)
normalized.append({"name": name, "url": url_value})
elif url is not None:
if isinstance(url, str):
raw = [url]
else:
raw = list(url)
for index, repository_url in enumerate(raw):
name = "origin" if index == 0 else f"repo{index}"
normalized.append({"name": name, "url": repository_url})
else:
raise ValueError("GitClone requires either 'url' or 'remotes'")
if len(normalized) == 0:
raise ValueError("At least one repository URL is required")
self.remotes = normalized
# Derived view used by repo_path keying and any introspection.
self.urls = [r["url"] for r in normalized]
self.repo_path = repo_path
self.branchNamePrefix = branchNamePrefix
self.credEnvPrefix = credEnvPrefix
@@ -36,33 +82,36 @@ class GitClone(AbstractStep[str]):
return url
def progress(self) -> str:
repo_url = self.url
self.repo_path = self.repo_path or os.path.join(tempfile.gettempdir(),
f"{self.env.id}_{hashlib.sha1(repo_url.encode()).hexdigest()[:5]}")
logger.info(f"Cloning repository {repo_url} to {self.repo_path}")
f"{self.env.id}_{hashlib.sha1('|'.join(self.urls).encode()).hexdigest()[:5]}")
logger.info(f"Cloning repositories {self.remotes} to {self.repo_path}")
os.makedirs(self.repo_path, exist_ok=True)
try:
if os.path.exists(os.path.join(self.repo_path, ".git")):
logger.info(f"Repository already cloned at {self.repo_path}, fetching updates.")
repo = git.Repo(self.repo_path)
if 'origin' not in repo.remotes or repo.remotes.origin.url != self._get_auth_git_url(repo_url):
repo.create_remote('origin', self._get_auth_git_url(repo_url))
result = repo.remotes.origin.fetch(prune=True)
if not result or any(fetch_info.flags & fetch_info.ERROR for fetch_info in result):
raise BaseException(f"Failed to fetch updates for {repo_url}")
else:
repo = git.Repo.init(self.repo_path)
repo.remotes.append(repo.create_remote(
'origin',
self._get_auth_git_url(repo_url)
))
if self.branchNamePrefix != "":
repo.config_writer().set_value('remote "origin"', "fetch",
f"+refs/heads/{self.branchNamePrefix}/*:refs/remotes/origin/{self.branchNamePrefix}/*").release()
repo.remotes.origin.fetch(prune=True)
if not os.path.exists(os.path.join(self.repo_path, ".git")):
raise BaseException(f"Failed to clone repository {repo_url} to {self.repo_path}")
for entry in self.remotes:
remote_name = entry["name"]
repository_url = entry["url"]
auth_url = self._get_auth_git_url(repository_url)
if remote_name in repo.remotes:
remote = repo.remotes[remote_name]
if remote.url != auth_url:
remote.set_url(auth_url)
else:
remote = repo.create_remote(remote_name, auth_url)
if remote_name == "origin" and self.branchNamePrefix != "":
with repo.config_writer() as config:
config.set_value('remote "origin"', "fetch",
f"+refs/heads/{self.branchNamePrefix}/*:refs/remotes/origin/{self.branchNamePrefix}/*")
result = remote.fetch(prune=True)
if not result or any(fetch_info.flags & fetch_info.ERROR for fetch_info in result):
raise BaseException(f"Failed to fetch updates for {repository_url}")
if not os.path.exists(os.path.join(self.repo_path, ".git")):
raise BaseException(f"Failed to clone repositories {self.remotes} to {self.repo_path}")
except BaseException as e:
logger.error(f"Error during git clone/fetch, removing directory {self.repo_path}: {str(e)}")
shutil.rmtree(self.repo_path)
@@ -73,21 +122,24 @@ class GitClone(AbstractStep[str]):
repo = git.Repo(self.repo_path)
result: Dict[str, List[Any]] = {}
for ref in repo.refs:
if ref.name.startswith('origin/') and not ref.name.startswith('origin/HEAD'):
branch_name = ref.name[len('origin/'):]
if not branch_name.startswith('auto/'): # Skip auto branches
result[branch_name] = []
cmt = [ref.commit]
for _ in range(10):
for commit in cmt:
result[branch_name].append({
'hexsha': commit.hexsha,
'author': commit.author.name,
'date': commit.committed_datetime.isoformat(),
'message': commit.message.strip()
})
if not ref.is_remote() or ref.name.endswith('/HEAD'):
continue
remote_name, branch_name = ref.name.split('/', 1)
if branch_name.startswith('auto/'): # Skip auto branches
continue
branch_key = branch_name if remote_name == 'origin' else ref.name
result[branch_key] = []
cmt = [ref.commit]
for _ in range(10):
for commit in cmt:
result[branch_key].append({
'hexsha': commit.hexsha,
'author': commit.author.name,
'date': commit.committed_datetime.isoformat(),
'message': commit.message.strip()
})
cmt = [p for c in cmt for p in c.parents]
cmt = [p for c in cmt for p in c.parents]
return result
@@ -112,6 +164,18 @@ def ensure_clean(repo: git.Repo) -> None:
raise BaseException(f"Changes in repo: U{repo.untracked_files}")
def _resolve_branch_ref(repo: git.Repo, branch_key: str, default_remote: str) -> str:
# Branch keys can be "branch", "origin/branch", "gitea_admin/branch", etc.
# If the key has a remote prefix that matches a known remote, use it
# verbatim; otherwise fall back to the primary configured remote so
# configs that don't name a remote `origin` still work.
if "/" in branch_key:
remote_name, _ = branch_key.split("/", 1)
if remote_name in repo.remotes:
return branch_key
return f"{default_remote}/{branch_key}"
class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
wd: GitClone
@@ -146,11 +210,19 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
def find_desired_commits(self, repo: git.Repo, branches: List[Tuple[str, str]]) -> Dict[Commit, str]:
# Extract commit ids for the selected branches
commit_ids: Dict[Commit, str] = {}
# Default remote: prefer the first remote GitClone was configured with
# (so non-"origin" configs like github/gitea_admin work without the
# caller having to qualify every branch key).
default_remote = "origin"
if isinstance(getattr(self.wd, "remotes", None), list) and self.wd.remotes:
primary = self.wd.remotes[0]["name"]
if primary in repo.remotes:
default_remote = primary
for branch_pair in branches:
try:
branch_name, desired_commit = branch_pair
if desired_commit == 'HEAD':
commit = repo.commit(f'origin/{branch_name}')
commit = repo.commit(_resolve_branch_ref(repo, branch_name, default_remote))
else:
commit = repo.commit(desired_commit)
commit_ids[commit] = branch_name
@@ -229,9 +301,10 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
version = '-'.join([x.hexsha[0:8] for x in sorted_commits])
for ref in repo.refs:
if ref.is_remote() and ref.commit == repo.head.commit and ref.name != 'origin/HEAD':
if ref.is_remote() and ref.commit == repo.head.commit and not ref.name.endswith('/HEAD'):
logger.info(f"Merge commit {ref.commit} corresponds to branch {ref}")
remote_branch_name = ref.name[len('origin/'):]
remote_name, branch_name = ref.name.split('/', 1)
remote_branch_name = branch_name if remote_name == 'origin' else ref.name
break
if self.push and remote_branch_name is None:
auto_branch_hash = hashlib.sha1(''.join([x.hexsha for x in sorted_commits]).encode()).hexdigest()
@@ -292,10 +365,9 @@ class GitUnmerge(AbstractStep[GitUnmergeResult]):
commits: List[Tuple[str, str]] = []
for v in version_parts:
commit = repo.commit(v)
# branches = [head.name for head in repo.heads if head.commit.hexsha == commit.hexsha]
branches = [ref.name for ref in repo.remotes.origin.refs if ref.commit.hexsha == commit.hexsha]
branches = [b[len('origin/'):] for b in branches if
b.startswith('origin/') and not b.startswith('origin/HEAD')]
branches = [ref.name for ref in repo.refs
if ref.is_remote() and not ref.name.endswith('/HEAD') and ref.commit.hexsha == commit.hexsha]
branches = [b[len('origin/'):] if b.startswith('origin/') else b for b in branches]
if len(branches) == 0 or branches[0].startswith('auto/'):
if childs is None:
@@ -304,10 +376,9 @@ class GitUnmerge(AbstractStep[GitUnmergeResult]):
while len(commitsSet) > 0:
current = commitsSet.pop()
for child in childs.get(current, []):
branches = [ref.name for ref in repo.remotes.origin.refs if
ref.commit.hexsha == child.hexsha]
branches = [b[len('origin/'):] for b in branches if
b.startswith('origin/') and not b.startswith('origin/HEAD')]
branches = [ref.name for ref in repo.refs
if ref.is_remote() and not ref.name.endswith('/HEAD') and ref.commit.hexsha == child.hexsha]
branches = [b[len('origin/'):] if b.startswith('origin/') else b for b in branches]
if len(branches) > 0:
commitsSet = set()
+5 -2
View File
@@ -42,8 +42,11 @@ class SharedStateHolderInMemory(AbstractStep[SharedState], SharedStateHolder):
def progress(self) -> SharedState:
if len(self.state.branches) == 0:
if self.unmerge:
unmerge = self.unmerge.progress()
self.state = replace(self.state, branches=unmerge.branches, token=uuid.uuid4().hex)
try:
unmerge = self.unmerge.progress()
self.state = replace(self.state, branches=unmerge.branches, token=uuid.uuid4().hex)
except BaseException as e:
logger.error("Unmerge failed: %s", e)
return self.state
+31
View File
@@ -0,0 +1,31 @@
import json
from dataclasses import asdict
import os
from typing import Any
class DataclassJSONEncoder(json.JSONEncoder):
def default(self, o: Any) -> Any:
if hasattr(o, '__dataclass_fields__'):
return asdict(o)
if isinstance(o, BaseException):
return str(o)
try:
return super().default(o)
except TypeError:
return str(o)
def custom_json_dumps(obj: Any) -> str:
return json.dumps(obj, cls=DataclassJSONEncoder)
def sigchld_handler(signum: int, frame: Any) -> None:
"""Reap zombie processes."""
while True:
try:
pid, status = os.waitpid(-1, os.WNOHANG)
if pid == 0:
break
except ChildProcessError:
break
+239
View File
@@ -0,0 +1,239 @@
import asyncio
import json
import logging
import os
import traceback
from contextlib import asynccontextmanager
from typing import Any, AsyncIterator, Dict, List, Optional, TypeVar
from fastapi import FastAPI, Response, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse
from app import App
from enironment import SharedStateConflictError, get_step
from utils import custom_json_dumps
from processing import reset_caches
from secondary import SecondaryManager
from steps.step import CachingStep
logger = logging.getLogger(__name__)
FRONTEND_DIR = os.path.join(os.path.dirname(__file__), '../frontend')
T = TypeVar('T')
def _merge_dicts(a: Dict[str, T], b: Dict[str, T]) -> Dict[str, T]:
result: Dict[str, T] = {}
for k in (a.keys() | b.keys()):
if k not in a or k not in b:
result[k] = a.get(k, b.get(k)) # type: ignore[assignment]
continue
if (
isinstance(a[k], dict)
and isinstance(b[k], dict)
):
result[k] = _merge_dicts(a[k], b[k]) # type: ignore[arg-type,assignment]
else:
result[k] = b[k]
return result
class WebApp:
def __init__(self, core: App, port: int) -> None:
self.core = core
self.port = port
self.ws_connections: Dict[WebSocket, Dict[str, Any]] = {}
self._event_loop: Optional[asyncio.AbstractEventLoop] = None
self.secondaryManager: Optional[SecondaryManager] = None
@asynccontextmanager
async def lifespan(fastapi_app: FastAPI) -> AsyncIterator[None]:
self._event_loop = asyncio.get_event_loop()
self.secondaryManager = SecondaryManager(
urls=os.getenv("SECONDARY_BRENCHER", ""),
on_update=lambda: self.broadcast_all(),
)
yield
self.app = FastAPI(lifespan=lifespan)
self.app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
self.app.get("/")(self.serve_index)
self.app.get("/state")(self.serve_state)
self.app.get("/branches")(self.serve_branches_route)
self.app.get("/{path:path}")(self.serve_static)
self.app.websocket("/ws")(self.websocket_endpoint)
# --- State assembly (local + secondary) ---
def get_global_envs_to_emit(self) -> Any:
local_envs = self.core.get_local_envs_to_emit()
merged: Dict[str, Any] = local_envs
for connector in self.secondaryManager or []:
v = connector.environments
common_keys = set(merged.keys()) & set(v.keys())
if len(common_keys) > 0:
self._schedule_async(self.broadcast_error({'message': f"Conflict: both master and secondary have environment with id {common_keys}"}))
merged = _merge_dicts(merged, v)
return merged
def get_global_branches_to_emit(self) -> Dict[str, Dict[str, List[Any]]]:
local_branches: Dict[str, Dict[str, List[Any]]] = self.core.get_local_branches_to_emit()
merged: Dict[str, Dict[str, List[Any]]] = local_branches
for connector in self.secondaryManager or []:
merged = _merge_dicts(merged, connector.branches)
return merged
# --- Async scheduling from sync threads ---
def _schedule_async(self, coro: Any) -> None:
if self._event_loop is not None and self._event_loop.is_running():
asyncio.run_coroutine_threadsafe(coro, self._event_loop)
# --- Broadcasting ---
async def broadcast(self, event: str, data: Any) -> None:
"""Broadcast a message to all connected WebSocket clients that have not seen this data yet."""
disconnected = set()
message = custom_json_dumps({event: data})
for websocket in self.ws_connections.keys():
if self.ws_connections[websocket].get(event) == message:
continue
try:
await websocket.send_text(message)
self.ws_connections[websocket][event] = message
except Exception as e:
logger.error(f"Error sending to websocket: {e}")
disconnected.add(websocket)
for ws in disconnected:
self.ws_connections.pop(ws, None)
async def broadcast_all(self) -> None:
await self.broadcast("branches", self.get_global_branches_to_emit())
await self.broadcast("environments", self.get_global_envs_to_emit())
async def broadcast_branches(self, data: Any) -> None:
await self.broadcast("branches", data)
async def broadcast_environments(self, data: Any) -> None:
await self.broadcast("environments", data)
await self.broadcast("version", self.get_version())
async def broadcast_error(self, data: Any) -> None:
await self.broadcast("error", data)
def emit_envs(self) -> None:
"""Sync callback used by App's processing thread to push updates to clients."""
self._schedule_async(self.broadcast_branches(self.get_global_branches_to_emit()))
self._schedule_async(self.broadcast_environments(self.get_global_envs_to_emit()))
# --- Routes ---
def get_version(self) -> str:
return os.getenv('VERSION', 'unknown') or 'unknown'
async def serve_index(self) -> FileResponse:
return FileResponse(os.path.join(FRONTEND_DIR, 'index.html'))
async def serve_state(self) -> Response:
state = self.get_global_envs_to_emit()
state['version'] = self.get_version()
return Response(content=custom_json_dumps(state), media_type="application/json")
async def serve_branches_route(self) -> Response:
return Response(content=custom_json_dumps(self.get_global_branches_to_emit()), media_type="application/json")
async def serve_static(self, path: str) -> FileResponse:
file_path = os.path.join(FRONTEND_DIR, path)
if os.path.exists(file_path) and os.path.isfile(file_path):
return FileResponse(file_path)
return FileResponse(os.path.join(FRONTEND_DIR, 'index.html'))
async def websocket_endpoint(self, websocket: WebSocket) -> None:
await websocket.accept()
self.ws_connections[websocket] = {}
try:
# Send initial state on connect and record it so duplicate broadcasts are suppressed
branches_payload = custom_json_dumps({"branches": self.get_global_branches_to_emit()})
envs_payload = custom_json_dumps({"environments": self.get_global_envs_to_emit()})
version_payload = custom_json_dumps({"version": self.get_version()})
await websocket.send_text(version_payload)
await websocket.send_text(branches_payload)
self.ws_connections[websocket]["branches"] = branches_payload
await websocket.send_text(envs_payload)
self.ws_connections[websocket]["environments"] = envs_payload
while True:
data = await websocket.receive_text()
message = json.loads(data)
if "update" in message:
update_data = message.get("update") or {}
logger.info(f"Received environment update: {update_data}")
if self.secondaryManager:
await self.secondaryManager.send({"update": update_data})
id = update_data.get('id', '')
if id == '':
reset_caches(list(self.core.environments.values()))
elif id not in self.core.environments.keys() and id not in {j for it in self.secondaryManager or [] for j in it.environments.keys()}:
logger.warning(f"Received update for unknown environment id {id}")
continue
elif id in self.core.environments.keys():
env = self.core.environments.get(id, None)
expected_token = update_data.get('token', '')
try:
if 'branches' in update_data:
if not env:
raise RuntimeError(f"Unknown env {update_data.get('id', '')}")
env.state.set_branches(update_data.get('branches', []), expected_token=expected_token)
if 'dry' in update_data:
if not env:
raise RuntimeError(f"Unknown env {update_data.get('id', '')}")
env.state.set_dry(bool(update_data['dry']), expected_token)
except SharedStateConflictError as conflict:
# if env:
# current_state = env.state.progress()
# await self.broadcast_error({
# 'code': 'BRANCH_STATE_CONFLICT',
# 'envId': id,
# 'message': str(conflict),
# 'current_state': {
# 'branches': current_state.branches,
# 'token': current_state.token,
# },
# })
continue
if env:
p = get_step(env.pipeline, type(env.state))
if isinstance(p, CachingStep):
p.reset()
logger.info(f"Updated environment {env.id} branches to {update_data.get('branches')}, dry={update_data.get('dry')}")
self.core.environment_update_event.set()
await self.broadcast_environments(self.get_global_envs_to_emit())
except WebSocketDisconnect:
self.ws_connections.pop(websocket, None)
except Exception as e:
logger.error(f"WebSocket error: {e}:{traceback.format_exc()}")
await self.broadcast_error({'message': f'{e}:{traceback.format_exc()}'})
# --- Run ---
def start(self) -> None:
import uvicorn
uvicorn.run(self.app, host='0.0.0.0', port=self.port, log_level='info')
+2 -1
View File
@@ -16,7 +16,8 @@ services:
- PYTHONUNBUFFERED=1
- VIRTUAL_HOST=brencher.rudolf.keenetic.link
- LETSENCRYPT_HOST=brencher.rudolf.keenetic.link
- PROFILES=brencher2,torrserv_proxy,immich,registry,gmail_mcp
- VERSION=${version}
- PROFILES=brencher2,torrserv_proxy,immich,gmail_mcp,hermes,hermes_infra,gitea
- SECONDARY_BRENCHER=http://host.docker.internal:5002
deploy:
replicas: 1
+24 -3
View File
@@ -13,6 +13,7 @@ const closeStatus = document.getElementById('close-status');
const refreshBranchesBtn = document.getElementById('refresh-branches');
const applyChangesBtn = document.getElementById('apply-changes');
const branchFilter = document.getElementById('branch-filter');
const appVersion = document.getElementById('app-version');
// Branches list (flattened) for UI rendering: [{ envId, envName, branch, commits: [] }]
let branches = [];
@@ -137,9 +138,20 @@ function escapeHtml(str) {
}
let renderedBranches = []
let renderedDryRunState = {}
let renderedColumnsSignature = ''
function getColumnsSignature() {
return JSON.stringify(columnsByEnv);
}
function renderBranches() {
if (JSON.stringify(filteredBranches) === JSON.stringify(renderedBranches) && JSON.stringify(dryRunByEnv) === JSON.stringify(renderedDryRunState)) {
const columnsSignature = getColumnsSignature();
if (
JSON.stringify(filteredBranches) === JSON.stringify(renderedBranches) &&
JSON.stringify(dryRunByEnv) === JSON.stringify(renderedDryRunState) &&
columnsSignature === renderedColumnsSignature
) {
return
}
checkForPendingChanges();
@@ -252,8 +264,9 @@ function renderBranches() {
filterBranches();
};
});
renderedBranches = filteredBranches;
renderedBranches = [...filteredBranches];
renderedDryRunState = {...dryRunByEnv};
renderedColumnsSignature = columnsSignature;
}
function updateBranchCommit(envId, branch, commit) {
@@ -465,14 +478,20 @@ function setupWebSockets() {
filterBranches();
renderJobs();
showStatus('Environments updated.');
} else if ('version' in message) {
appVersion.textContent = `Version: ${message.version || 'unknown'}`;
} else if ('error' in message) {
console.error('Error from server:', message.error);
if (message.error && message.error.code === 'BRANCH_STATE_CONFLICT' && message.error.envId) {
const envId = message.error.envId;
const current = message.error.current_state || {};
const hadLocalSelection = Object.prototype.hasOwnProperty.call(selectedBranchesByEnv, envId);
if (Array.isArray(current.branches)) {
selectedBranchesByEnv[envId] = [...current.branches];
serverSelectedBranchesByEnv[envId] = [...current.branches];
// Preserve the user's pending edits if they already changed selection locally.
if (!hadLocalSelection) {
selectedBranchesByEnv[envId] = [...current.branches];
}
}
if (typeof current.token === 'string') {
serverTokenByEnv[envId] = current.token;
@@ -503,6 +522,8 @@ function updateEnvironment() {
if (JSON.stringify(localSel) !== JSON.stringify(serverSel)) {
if (ws && ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify({ update: { id: envId, branches: selectedBranchesByEnv[envId], token: serverTokenByEnv[envId] } }));
// Keep pending-state tracking aligned with what we just submitted.
serverSelectedBranchesByEnv[envId] = [...(selectedBranchesByEnv[envId] || [])];
}
branchFilter.value = '';
}
+1
View File
@@ -10,6 +10,7 @@
<div class="container">
<header>
<h1>Brencher - Branch Merger</h1>
<p class="version" id="app-version">Version: loading...</p>
<div id="status-bar" class="status-bar hidden">
<span id="status-message"></span>
<button id="close-status">×</button>
+7
View File
@@ -15,6 +15,13 @@ body {
padding: 32px;
}
.version {
color: #6c757d;
font-family: monospace;
font-size: 0.85rem;
margin-top: -12px;
}
.primary-button {
background: #007bff;
color: #fff;
+6
View File
@@ -5,8 +5,14 @@ pytest-asyncio
playwright
pytest-playwright
mypy
flake8
ty
types-PyYAML
types-requests
types-docker
testcontainers
+54
View File
@@ -0,0 +1,54 @@
from enironment import Environment
from steps.docker_plain import DockerImageBuild, DockerContainerCheck, DockerContainerDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
container_check = DockerContainerCheck(
container_name="test_plain-container"
)
unmerge = GitUnmerge(wd=clone, check=container_check)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="test_bot"
)
# Step 1: Build the image
image_build = DockerImageBuild(
wd=checkoutMerged,
dockerfile_path="Dockerfile_test_app",
image_name="test_plain",
image_tag=lambda: "auto-" + checkoutMerged.progress().version,
nocache=False,
)
# Step 3: Deploy the container
container_deploy = DockerContainerDeploy(
image_build=image_build,
container_name="test_plain-container",
ports={"5001/tcp": 5002},
environment={"PROFILES": "no_profiles"},
restart_policy={"Name": "unless-stopped"},
)
__all__ = ["test_local1"]
test_local1 = Environment(
id="test_local1",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
image_build,
container_deploy,
container_deploy,
unmerge
]
)
+25 -11
View File
@@ -4,9 +4,10 @@ Pytest configuration and shared fixtures for integration tests.
This file configures the Python path to allow importing backend modules
and provides shared fixtures for all tests.
"""
from typing import Generator, Callable, Protocol
from typing import Any, Generator, Callable, Protocol, TypeVar
import pytest
import requests
from tests.test_remote_repo import RemoteRepoHelper
@@ -26,25 +27,38 @@ import pytest
class EventuallyFn(Protocol):
def __call__(
self,
assert_fn: Callable[[], bool],
timeout: float = 2.0,
interval: float = 0.05,
assert_fn: Callable[[], None],
timeout: float = 5.0,
interval: float = 0.5,
) -> None: ...
T = TypeVar('T')
def assert_equal(a: T, b:T, message: str) -> None:
if a != b:
raise AssertionError(f"{message}: {a} != {b}")
def check_state(url: str, predicate: Callable[[Any], None]) -> None:
response = requests.get(f"{url}/state", timeout=5)
assert response.status_code == 200, f"Expected status code 200, got {response.status_code}"
state_data = response.json()
try:
predicate(state_data)
except Exception as e:
raise AssertionError(f"Predicate check failed with error: {e}\nState data: {state_data}") from e
@pytest.fixture
def eventually() -> EventuallyFn:
def _eventually(assert_fn: Callable[[], bool], timeout: float = 2.0, interval: float = 0.05) -> None:
def _eventually(assert_fn: Callable[[], None], timeout: float = 5.0, interval: float = 0.5) -> None:
deadline = time.monotonic() + timeout
last_error = None
last_error: BaseException | None = None
while time.monotonic() < deadline:
try:
res = assert_fn()
if res:
return
except AssertionError as err:
assert_fn()
return
except Exception as err:
last_error = err
time.sleep(interval)
time.sleep(interval)
raise AssertionError(f"Condition not met within {timeout:.2f}s") from last_error
+211
View File
@@ -0,0 +1,211 @@
import logging
from contextlib import contextmanager
from typing import Generator
from playwright.sync_api import Locator, Page, sync_playwright
logger = logging.getLogger(__name__)
class BrencherPage:
def __init__(self, page: Page, env_id: str, action_logger: logging.Logger) -> None:
self._page = page
self._env_id = env_id
self._logger = action_logger
def _dry_run_button(self) -> Locator:
button = self._page.locator(f'h3:has-text("{self._env_id}") button.dry-run-btn')
button.wait_for(state="visible", timeout=10000)
return button
def _apply_changes_button(self) -> Locator:
button = self._page.locator("#apply-changes")
button.wait_for(state="visible", timeout=10000)
return button
def _branch_checkbox(self, branch_name: str) -> Locator:
checkbox = self._page.locator(
f'input[type="checkbox"][data-env="{self._env_id}"][value="{branch_name}"]'
)
checkbox.wait_for(state="visible", timeout=10000)
return checkbox
def _desired_commit_select(self, branch_name: str) -> Locator:
select = self._page.locator(
f'select.desired-commit[data-env="{self._env_id}"][data-branch="{branch_name}"]'
)
select.wait_for(state="visible", timeout=10000)
return select
def verify_dry_run_on(self) -> None:
btn_title = self._dry_run_button().get_attribute("title") or ""
assert "Dry run on" in btn_title, f"Expected dry run to be active, got title: {btn_title!r}"
self._logger.info(f"Dry-run button title before click: {btn_title!r}")
def set_dry_run_off(self) -> None:
self._dry_run_button().click()
self._logger.info("Clicked dry-run button to disable dry run")
def set_branch_unselected(self, branch_name: str) -> None:
checkbox = self._branch_checkbox(branch_name)
if checkbox.is_checked():
checkbox.uncheck()
self._logger.info("Unchecked branch '%s'", branch_name)
def set_branch_desired_commit(self, branch_name: str, commit_hash: str) -> None:
self._desired_commit_select(branch_name).select_option(value=commit_hash)
self._logger.info("Set desired commit for branch '%s' to '%s'", branch_name, commit_hash)
def set_apply_changes(self) -> None:
self._apply_changes_button().click()
self._logger.info("Clicked Apply Changes")
def set_simulate_branch_state_conflict(self) -> None:
self._page.evaluate(
"""({ envId }) => {
const currentToken =
(typeof serverTokenByEnv !== 'undefined' && serverTokenByEnv[envId])
? serverTokenByEnv[envId]
: 'server-token';
const staleToken = `${currentToken}-stale`;
if (typeof serverTokenByEnv !== 'undefined') {
// Simulate stale client-side token so next apply uses an outdated value.
serverTokenByEnv[envId] = staleToken;
}
}""",
{
"envId": self._env_id,
},
)
self._logger.info("Set stale branch token for '%s'", self._env_id)
def click_refresh(self) -> None:
self._page.locator("#refresh-branches").click()
self._logger.info("Clicked refresh button")
def verify_dry_run_off(self) -> None:
self._page.wait_for_function(
"""(targetEnvId) => {
const h3s = [...document.querySelectorAll('h3.env-title')];
const h3 = h3s.find(el => el.textContent.includes(targetEnvId));
if (!h3) return false;
const btn = h3.querySelector('button.dry-run-btn');
return btn && !btn.title.includes('Dry run on');
}""",
arg=self._env_id,
timeout=10000,
)
def verify_apply_changes_visible(self) -> None:
self._page.wait_for_function(
"""() => {
const btn = document.querySelector('#apply-changes');
return !!btn && !btn.classList.contains('hidden');
}""",
timeout=10000,
)
self._logger.info("Apply Changes button is visible")
def verify_apply_changes_hidden(self) -> None:
self._page.wait_for_function(
"""() => {
const btn = document.querySelector('#apply-changes');
return !!btn && btn.classList.contains('hidden');
}""",
timeout=10000,
)
self._logger.info("Apply Changes button is hidden")
def verify_branch_column_value(self, branch_name: str, column_name: str, expected_value: str, timeout_ms: int = 10000) -> None:
self._page.wait_for_function(
"""({ envId, branchName, columnName, expectedValue }) => {
const row = document.querySelector(`tr.branch-row[data-env="${envId}"][data-branch="${branchName}"]`);
if (!row) return false;
const table = row.closest('table.branches-table');
if (!table) return false;
const headers = [...table.querySelectorAll('thead th')].map(th => (th.textContent || '').trim());
const colIdx = headers.indexOf(columnName);
if (colIdx === -1) return false;
const cells = [...row.querySelectorAll('td')];
if (cells.length <= colIdx) return false;
return ((cells[colIdx].textContent || '').trim() === expectedValue);
}""",
arg={
"envId": self._env_id,
"branchName": branch_name,
"columnName": column_name,
"expectedValue": expected_value,
},
timeout=timeout_ms,
)
self._logger.info(
"Verified column '%s' for branch '%s' equals '%s'",
column_name,
branch_name,
expected_value,
)
def verify_pipeline_step_status(self, step_name: str, expected_status: str, timeout_ms: int = 10000) -> None:
"""
Verify that a pipeline step reaches the expected status by polling the page DOM.
Args:
step_name: Name of the pipeline step (e.g., "DockerContainerDeploy")
expected_status: Expected status value ("running", "ok", "error", or "dry-run")
timeout_ms: Timeout in milliseconds (default 10000)
"""
# JavaScript to check pipeline step status
js_check = """(stepName, expectedStatus) => {
const jobItems = document.querySelectorAll('.job-item');
for (const item of jobItems) {
const headerText = item.querySelector('.job-header')?.textContent || '';
if (headerText.includes(stepName)) {
const spinner = item.querySelector('.step-spinner');
const errorIndicator = item.querySelector('span[title="Error"]');
if (expectedStatus === 'running') {
return spinner !== null;
} else if (expectedStatus === 'ok') {
return spinner === null && errorIndicator === null;
} else if (expectedStatus === 'error') {
return errorIndicator !== null;
} else if (expectedStatus === 'dry-run') {
// For dry-run, check the spoiler content
const spoiler = item.querySelector('.job-spoiler');
if (spoiler && spoiler.style.display !== 'none') {
const statusText = spoiler.textContent;
return statusText.includes('dry-run') || statusText.includes('dry_run');
}
return false;
}
}
}
return false;
}"""
self._page.wait_for_function(
js_check,
arg=[step_name, expected_status],
timeout=timeout_ms,
)
self._logger.info(f"Pipeline step '{step_name}' reached status '{expected_status}'")
@contextmanager
def brencher_page(
app_link: str,
env_id: str,
action_logger: logging.Logger | None = None,
) -> Generator[BrencherPage, None, None]:
active_logger = action_logger or logger
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=True)
page = browser.new_page()
page.goto(app_link)
page.wait_for_load_state("networkidle")
try:
yield BrencherPage(page, env_id, active_logger)
finally:
browser.close()
+127
View File
@@ -0,0 +1,127 @@
import logging
import threading
import time
from typing import Any, Generator
import docker
import pytest
import requests
import socket
from app import App
from tests.configs import nginx
from tests.conftest import EventuallyFn, assert_equal
from tests.playwright_helper import brencher_page
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
APP_PORT = 5003
APP_URL = f"http://localhost:{APP_PORT}"
CONTAINER_NAME = "test_plain-container"
# Module-level reference to server thread to ensure proper cleanup
_server_thread: threading.Thread | None = None
def _is_port_free(port: int, timeout: float = 5.0) -> bool:
"""Check if a port is available for binding."""
deadline = time.time() + timeout
while time.time() < deadline:
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
result = sock.connect_ex(('127.0.0.1', port))
sock.close()
if result != 0:
return True
except Exception:
return True
time.sleep(0.1)
return False
class TestApplyChangesPlaywright:
@pytest.fixture(autouse=True)
def setup_teardown(self) -> Generator[None, Any, None]:
global _server_thread
def stop_and_remove_container() -> None:
client = docker.from_env()
try:
container = client.containers.get(CONTAINER_NAME)
container.stop(timeout=10)
container.remove(force=True)
logger.info("Container %s stopped and removed", CONTAINER_NAME)
except Exception:
logger.info("Container %s not found, no need to stop/remove", CONTAINER_NAME)
# Ensure port is free before starting
_is_port_free(APP_PORT, timeout=5.0)
stop_and_remove_container()
yield None
# Clean up server thread
if _server_thread and _server_thread.is_alive():
_server_thread.join(timeout=2.0)
logger.info("Waited for server thread to finish")
# Ensure port is free after test
_is_port_free(APP_PORT, timeout=5.0)
stop_and_remove_container()
_server_thread = None
def test_apply_button_hides_after_apply_branch_change(self, eventually: EventuallyFn) -> None:
global _server_thread
# Start app with one pre-selected branch so unchecking creates pending changes.
env = nginx.test_local1
env.state.set_branches([("test/main", "HEAD")])
env.state.set_dry(True)
app = App({env.id: env})
_server_thread = threading.Thread(target=lambda: app.runWeb(APP_PORT), daemon=True)
_server_thread.start()
eventually(
lambda: assert_equal(
requests.get(f"{APP_URL}/state", timeout=5).status_code,
200,
"Server did not respond with 200 OK within timeout",
)
)
with brencher_page(APP_URL, env.id) as page:
page.set_branch_unselected("test/main")
page.verify_apply_changes_visible()
page.set_apply_changes()
eventually(lambda: page.verify_apply_changes_hidden(), timeout=10, interval=0.5)
def test_apply_button_reappears_on_branch_conflict(self, eventually: EventuallyFn) -> None:
global _server_thread
env = nginx.test_local1
env.state.set_branches([("test/main", "HEAD")])
env.state.set_dry(True)
app = App({env.id: env})
_server_thread = threading.Thread(target=lambda: app.runWeb(APP_PORT), daemon=True)
_server_thread.start()
eventually(
lambda: assert_equal(
requests.get(f"{APP_URL}/state", timeout=5).status_code,
200,
"Server did not respond with 200 OK within timeout",
)
)
with brencher_page(APP_URL, env.id) as page:
page.set_branch_unselected("test/main")
page.verify_apply_changes_visible()
# Set stale token in client JS object so backend returns a real conflict on apply.
page.set_simulate_branch_state_conflict()
page.set_apply_changes()
eventually(lambda: page.verify_apply_changes_visible(), timeout=10, interval=0.5)
+4 -3
View File
@@ -8,7 +8,8 @@ import docker
import pytest
import requests
from app import App
from .conftest import EventuallyFn
from configs import brencher_local1
from .conftest import EventuallyFn, assert_equal
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
@@ -34,7 +35,7 @@ class TestDockerContainer:
async def test_start(self, eventually: EventuallyFn) -> None:
logger.info(f"Starting")
app = App(cli_env_ids_str="brencher_local1")
app = App({"brencher_local1": brencher_local1.brencher_local1})
# app.runWeb(5001)
processing = threading.Thread(target=lambda: app.runWeb(5001), daemon=True)
processing.start()
@@ -42,7 +43,7 @@ class TestDockerContainer:
# await asyncio.sleep(5000)
eventually(
lambda: requests.get("http://localhost:5001/state", timeout=5000).status_code == 200,
lambda: assert_equal(requests.get("http://localhost:5001/state", timeout=5000).status_code, 200, "Server did not respond with 200 OK within timeout"),
20.0,
1.0,
)
+182
View File
@@ -0,0 +1,182 @@
import logging
import socket
import threading
import time
from typing import Any, Generator
import docker
import pytest
import requests
from app import App
from tests.configs import nginx
from tests.conftest import EventuallyFn, assert_equal
from tests.playwright_helper import brencher_page
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
APP_PORT = 5004
APP_URL = f"http://localhost:{APP_PORT}"
CONTAINER_NAME = "test_plain-container"
TARGET_BRANCH = "test/main"
_server_thread: threading.Thread | None = None
def _is_port_free(port: int, timeout: float = 5.0) -> bool:
"""Check if a port is available for binding."""
deadline = time.time() + timeout
while time.time() < deadline:
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
result = sock.connect_ex(("127.0.0.1", port))
sock.close()
if result != 0:
return True
except Exception:
return True
time.sleep(0.1)
return False
def _get_state() -> dict[str, Any]:
response = requests.get(f"{APP_URL}/state", timeout=5)
assert response.status_code == 200, f"Expected status 200 from /state, got {response.status_code}"
data = response.json()
assert isinstance(data, dict), f"Expected dict from /state, got {type(data)}"
return data
def _get_deployed_short_hash(env_id: str, branch: str) -> str:
state = _get_state()
env_state = state.get(env_id)
assert isinstance(env_state, dict), f"No environment state for {env_id}: {state}"
pipeline = env_state.get("pipeline")
assert isinstance(pipeline, list), f"Pipeline is not list for {env_id}: {env_state}"
step = next((p for p in pipeline if p.get("name") == "GitUnmerge"), None)
assert isinstance(step, dict), f"GitUnmerge step not found in pipeline: {pipeline}"
assert not bool(step.get("is_running")), f"GitUnmerge is still running: {step}"
status = step.get("status")
assert isinstance(status, dict), f"GitUnmerge status is not an object: {status}"
columns = status.get("columns")
assert isinstance(columns, dict), f"GitUnmerge columns missing: {status}"
deployed = columns.get("Deployed")
assert isinstance(deployed, dict), f"Deployed column missing: {columns}"
value = deployed.get(branch)
assert isinstance(value, str) and len(value) == 8, f"Deployed value for {branch} invalid: {value}"
return value
def _pick_target_commit(env_id: str, branch: str) -> str:
response = requests.get(f"{APP_URL}/branches", timeout=5)
assert response.status_code == 200, f"Expected status 200 from /branches, got {response.status_code}"
data = response.json()
assert isinstance(data, dict), f"Expected dict from /branches, got {type(data)}"
env_branches = data.get(env_id, {})
assert isinstance(env_branches, dict), f"Expected env branch map for {env_id}, got {env_branches}"
commits = env_branches.get(branch, [])
assert isinstance(commits, list), f"Expected commits list for {branch}, got {commits}"
for commit in commits:
if not isinstance(commit, dict):
continue
hexsha = commit.get("hexsha")
if isinstance(hexsha, str) and len(hexsha) >= 8:
return hexsha
raise AssertionError(f"Unable to find any commit for branch {branch}.")
def _assert_deployed_hash_available(env_id: str, branch: str) -> None:
value = _get_deployed_short_hash(env_id, branch)
assert isinstance(value, str) and len(value) == 8
class TestDeployColumnPlaywright:
@pytest.fixture(autouse=True)
def setup_teardown(self) -> Generator[None, Any, None]:
global _server_thread
def stop_and_remove_container() -> None:
client = docker.from_env()
try:
container = client.containers.get(CONTAINER_NAME)
container.stop(timeout=10)
container.remove(force=True)
logger.info("Container %s stopped and removed", CONTAINER_NAME)
except Exception:
logger.info("Container %s not found, no need to stop/remove", CONTAINER_NAME)
_is_port_free(APP_PORT, timeout=5.0)
stop_and_remove_container()
yield None
if _server_thread and _server_thread.is_alive():
_server_thread.join(timeout=2.0)
logger.info("Waited for server thread to finish")
_is_port_free(APP_PORT, timeout=5.0)
stop_and_remove_container()
_server_thread = None
def test_deployed_column_updates_after_apply_changes_and_deploy(self, eventually: EventuallyFn) -> None:
global _server_thread
env = nginx.test_local1
env.state.set_branches([(TARGET_BRANCH, "HEAD")])
env.state.set_dry(True)
app = App({env.id: env})
_server_thread = threading.Thread(target=lambda: app.runWeb(APP_PORT), daemon=True)
_server_thread.start()
eventually(
lambda: assert_equal(
requests.get(f"{APP_URL}/state", timeout=5).status_code,
200,
"Server did not respond with 200 OK within timeout",
),
timeout=20,
interval=0.5,
)
target_commit = _pick_target_commit(env.id, TARGET_BRANCH)
with brencher_page(APP_URL, env.id) as page:
page.verify_dry_run_on()
page.set_branch_desired_commit(TARGET_BRANCH, target_commit)
page.verify_apply_changes_visible()
page.set_apply_changes()
eventually(lambda: page.verify_apply_changes_hidden(), timeout=20, interval=0.5)
page.set_dry_run_off()
eventually(lambda: page.verify_dry_run_off(), timeout=20, interval=0.5)
page.click_refresh()
eventually(
lambda: _assert_deployed_hash_available(env.id, TARGET_BRANCH),
timeout=90,
interval=1.0,
)
deployed_short_hash = _get_deployed_short_hash(env.id, TARGET_BRANCH)
eventually(
lambda: page.verify_branch_column_value(TARGET_BRANCH, "Deployed", deployed_short_hash),
timeout=30,
interval=0.5,
)
+92 -84
View File
@@ -1,121 +1,129 @@
import logging
import threading
from typing import Any, Generator
import time
import socket
from typing import Any, Callable, Generator, TypeVar
import docker
import pytest
import requests
from playwright.sync_api import Page, sync_playwright
from app import App
from .conftest import EventuallyFn
from .conftest import EventuallyFn, assert_equal, check_state
from .playwright_helper import brencher_page
from tests.configs import nginx
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
APP_PORT = 5003
APP_URL = f"http://localhost:{APP_PORT}"
ENV_ID = "brencher_local1"
CONTAINER_NAME = "brencher_plain-container"
CONTAINER_NAME = "test_plain-container"
# Module-level reference to server thread to ensure proper cleanup
_server_thread: threading.Thread | None = None
def _get_pipeline_step(state_data: Any, env_id: str, step_name: str) -> Any:
pipeline = state_data[env_id]["pipeline"]
for step in pipeline:
if step["name"] == step_name:
return step
return None
def _is_port_free(port: int, timeout: float = 5.0) -> bool:
"""Check if a port is available for binding."""
deadline = time.time() + timeout
while time.time() < deadline:
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
result = sock.connect_ex(('127.0.0.1', port))
sock.close()
if result != 0:
return True
except Exception:
return True
time.sleep(0.1)
return False
@pytest.mark.skip
class TestDryRunPlaywright:
@pytest.fixture(autouse=True)
def setup_teardown(self) -> Generator[None, Any, None]:
global _server_thread
def stop_and_remove_container() -> None:
client = docker.from_env()
try:
container = client.containers.get(CONTAINER_NAME)
container.stop(timeout=10)
container.remove(force=True)
logger.info(f"Container {CONTAINER_NAME} stopped and removed")
except Exception:
logger.info(f"Container {CONTAINER_NAME} not found, no need to stop/remove")
# Ensure port is free before starting
_is_port_free(APP_PORT, timeout=5.0)
stop_and_remove_container()
yield None
# Clean up server thread
if _server_thread and _server_thread.is_alive():
_server_thread.join(timeout=2.0)
logger.info("Waited for server thread to finish")
# Ensure port is free after test
_is_port_free(APP_PORT, timeout=5.0)
stop_and_remove_container()
_server_thread = None
client = docker.from_env()
containers = client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
for container in containers:
container.stop(timeout=10)
container.remove(force=True)
logger.info(f"Container {CONTAINER_NAME} stopped and removed")
def test_dry_run_prevents_deploy_until_disabled(self, eventually: EventuallyFn) -> None:
# Start app with brencher_local1 with main branch pre-selected.
# App starts with dry=True by default (SharedStateHolderInMemory default).
app = App(cli_env_ids_str=f"{ENV_ID}:main")
server_thread = threading.Thread(target=lambda: app.runWeb(APP_PORT), daemon=True)
server_thread.start()
global _server_thread
# Start app with nginx_local1 with main branch pre-selected and dry mode on.
env = nginx.test_local1
env.state.set_branches([("test/main", "HEAD")])
env.state.set_dry(True)
app = App({env.id: env})
_server_thread = threading.Thread(target=lambda: app.runWeb(APP_PORT), daemon=True)
_server_thread.start()
# Wait for server to be up
eventually(
lambda: requests.get(f"{APP_URL}/state", timeout=5).status_code == 200,
timeout=20.0,
interval=1.0,
)
eventually(lambda: assert_equal(requests.get(f"{APP_URL}/state", timeout=5).status_code, 200, "Server did not respond with 200 OK within timeout"))
logger.info("Server is up")
# Wait for the pipeline to stabilize with dry=True:
# DockerContainerDeploy should return status="dry-run"
def _deploy_is_dry_run() -> bool:
state_data = requests.get(f"{APP_URL}/state", timeout=5).json()
step = _get_pipeline_step(state_data, ENV_ID, "DockerContainerDeploy")
if step is None or step.get("error"):
return False
return step["status"].get("status") == "dry-run"
eventually(_deploy_is_dry_run, timeout=120.0, interval=2.0)
logger.info("Pipeline is in dry-run mode")
# Verify no real container was created
docker_client = docker.from_env()
real_containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(real_containers) == 0, "Container should NOT exist during dry run"
# Open browser and click the dry-run button to disable dry run
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page: Page = browser.new_page()
page.goto(APP_URL)
page.wait_for_load_state("networkidle")
# The dry-run button shows ⏸ when active (dry=True) and ▶ when disabled (dry=False).
# It is inside the h3 heading for brencher_local1.
dry_run_btn = page.locator(f'h3:has-text("{ENV_ID}") button.dry-run-btn')
dry_run_btn.wait_for(state="visible", timeout=10000)
# Verify button indicates dry-run is currently active (title contains "Dry run on")
btn_title = dry_run_btn.get_attribute("title") or ""
assert "Dry run on" in btn_title, f"Expected dry run to be active, got title: {btn_title!r}"
logger.info(f"Dry-run button title before click: {btn_title!r}")
# Click to disable dry run
dry_run_btn.click()
logger.info("Clicked dry-run button to disable dry run")
# Wait for button to update to "Running" state (title no longer says "Dry run on")
page.wait_for_function(
"""(envId) => {
const h3s = [...document.querySelectorAll('h3.env-title')];
const h3 = h3s.find(el => el.textContent.includes(envId));
if (!h3) return false;
const btn = h3.querySelector('button.dry-run-btn');
return btn && !btn.title.includes('Dry run on');
}""",
arg=ENV_ID,
timeout=10000,
with brencher_page(APP_URL, env.id) as page:
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] == "SharedStateHolderInMemory"
)["status"]['dry'], True, "Pipeline step status is not 'running'")),
)
browser.close()
# page.verify_pipeline_step_status("DockerContainerDeploy", "dry-run")
logger.info("Pipeline is in dry-run mode")
# Wait for pipeline to re-run and actually deploy the container
def _deploy_is_running() -> bool:
state_data = requests.get(f"{APP_URL}/state", timeout=5).json()
step = _get_pipeline_step(state_data, ENV_ID, "DockerContainerDeploy")
if step is None or step.get("error"):
return False
return step["status"].get("status") == "running"
# Verify no real container was created
docker_client = docker.from_env()
real_containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(real_containers) == 0, "Container should NOT exist during dry run"
eventually(_deploy_is_running, timeout=300.0, interval=5.0)
# Disable dry run from the UI.
page.verify_dry_run_on()
page.set_dry_run_off()
page.verify_dry_run_off()
page.click_refresh() # TODO Remove it
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] =="SharedStateHolderInMemory"
)["status"]['dry'], False, "Pipeline step status is not 'running'")),
)
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal({p["name"] for p in s[env.id]["pipeline"] if bool(p['is_running'])}, set(), "Pipeline still running")),
timeout=20,
interval=1.0,
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "running")
logger.info("Container deployed and running after dry run disabled")
# Final sanity: real container must exist and be running
+86 -1
View File
@@ -5,10 +5,12 @@ These tests run in a Docker-like isolated environment to avoid harming the host
They simulate remote and local git repositories and test various merge scenarios.
"""
import os
import shutil
import tempfile
import git
import pytest
from steps.git import CheckoutAndMergeResult, GitUnmergeResult
from steps.git import CheckoutAndMergeResult, GitClone, GitUnmergeResult
from .test_remote_repo import RemoteRepoHelper
@@ -16,6 +18,89 @@ from .test_remote_repo import RemoteRepoHelper
class TestGitIntegration:
"""Integration tests for git operations"""
def test_git_clone_multiple_urls(self, repo_helper: RemoteRepoHelper) -> None:
repo_helper.create_commit(repo_helper.repo, "master", "master", "README.md", "content", "Initial commit")
second_url = f"file://{repo_helper.remote_dir}"
clone = GitClone(url=[repo_helper.remote_dir, second_url])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
assert {remote.name for remote in repo.remotes} == {"origin", "repo1"}
assert repo.remotes.origin.url == repo_helper.remote_dir
assert repo.remotes.repo1.url == second_url
assert set(clone.get_branches()) == {"master", "repo1/master"}
def test_git_clone_named_remotes(self, repo_helper: RemoteRepoHelper) -> None:
# Two real local repos served as separate "remotes". The user controls
# their names explicitly via the `remotes=[{name,url}]` API.
def fresh_repo(prefix: str, message: str) -> str:
path = tempfile.mkdtemp(prefix=prefix)
repo = git.Repo.init(path, bare=False)
with repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
repo_helper.create_commit(repo, "master", "master", "README.md", "content", message)
return path
github_dir = fresh_repo("test_named_github_", "github initial")
gitea_dir = fresh_repo("test_named_gitea_", "gitea initial")
try:
clone = GitClone(remotes=[
{"name": "github", "url": github_dir},
{"name": "gitea_admin", "url": gitea_dir},
])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
# User-chosen names appear exactly; the auto-generated "origin" /
# "repo1" names from the positional-list form do NOT leak in.
assert {r.name for r in repo.remotes} == {"github", "gitea_admin"}
assert "origin" not in repo.remotes
assert "repo1" not in repo.remotes
assert repo.remotes.github.url == github_dir
assert repo.remotes.gitea_admin.url == gitea_dir
# `get_branches` keeps the <remote>/<branch> shape for non-origin
# names; the user gets the full provenance in the key.
assert set(clone.get_branches()) == {"github/master", "gitea_admin/master"}
finally:
shutil.rmtree(github_dir, ignore_errors=True)
shutil.rmtree(gitea_dir, ignore_errors=True)
def test_git_clone_remotes_validation(self) -> None:
# Empty list: rejected (would produce an unusable clone).
with pytest.raises(ValueError, match="At least one repository URL is required"):
GitClone(remotes=[])
# Missing / empty / non-string name or url: rejected.
with pytest.raises(ValueError, match="non-empty 'name'"):
GitClone(remotes=[{"url": "git@example.com:foo/bar.git"}])
with pytest.raises(ValueError, match="non-empty 'name'"):
GitClone(remotes=[{"name": "", "url": "git@example.com:foo/bar.git"}])
with pytest.raises(ValueError, match="non-empty 'url'"):
GitClone(remotes=[{"name": "github"}])
with pytest.raises(ValueError, match="non-empty 'url'"):
GitClone(remotes=[{"name": "github", "url": ""}])
# Duplicate name: rejected.
with pytest.raises(ValueError, match="Duplicate remote name"):
GitClone(remotes=[
{"name": "github", "url": "git@example.com:a/b.git"},
{"name": "github", "url": "git@example.com:c/d.git"},
])
# Passing both `url` and `remotes`: rejected.
with pytest.raises(ValueError, match="not both"):
GitClone(url="git@example.com:a/b.git", remotes=[{"name": "github", "url": "git@example.com:a/b.git"}])
# Passing neither: rejected.
with pytest.raises(ValueError, match="requires either 'url' or 'remotes'"):
GitClone()
def test_checkout_merged_one_branch(self, repo_helper: RemoteRepoHelper) -> None:
# Create main branch with initial commit
commit1 = repo_helper.create_commit(repo_helper.repo, "master", "master", "file1.txt", "content1",