Compare commits

...
Author SHA1 Message Date
admin dbcae04782 x
Python package / build (3.12.3) (pull_request) Successful in 2m19s
Python package / build (3.13) (pull_request) Successful in 2m20s
Python package / build (3.14) (pull_request) Successful in 2m30s
2026-10-11 19:32:35 +02:00
admin 56b6660c1f x
Python package / build (3.12.3) (pull_request) Successful in 2m22s
Python package / build (3.13) (pull_request) Successful in 2m19s
Python package / build (3.14) (pull_request) Successful in 2m25s
2026-10-11 17:20:09 +02:00
hermes 13d6d9f95d fix: push connectivity tester to org1
Python package / build (3.12.3) (pull_request) Successful in 2m46s
Python package / build (3.13) (pull_request) Successful in 3m27s
Python package / build (3.14) (pull_request) Successful in 2m34s
2026-10-10 12:33:22 +00:00
hermes 0290b5bbac fix: align connectivity tester registry namespace
Python package / build (3.12.3) (pull_request) Successful in 3m3s
Python package / build (3.13) (pull_request) Successful in 2m30s
Python package / build (3.14) (pull_request) Successful in 2m34s
2026-10-10 10:19:55 +00:00
hermes b313a39e23 feat: push auto branches for actions 2026-10-09 10:51:57 +00:00
hermes 4fca3a4856 feat: fetch connectivity tester forks
Python package / build (3.12.3) (pull_request) Successful in 2m21s
Python package / build (3.13) (pull_request) Successful in 2m25s
Python package / build (3.14) (pull_request) Successful in 2m33s
2026-10-08 20:18:01 +00:00
hermes bee03bdb52 feat: use dedicated connectivity tester repository
Python package / build (3.12.3) (pull_request) Canceled after 0s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
2026-10-08 20:01:52 +00:00
hermes 272a06a51c merge main: resolve connectivity tester conflicts
Python package / build (3.12.3) (pull_request) Successful in 2m43s
Python package / build (3.13) (pull_request) Successful in 3m16s
Python package / build (3.14) (pull_request) Successful in 2m10s
2026-09-29 17:57:30 +00:00
hermes bf4533c0fe fix(deploy): forward registry auth to swarm nodes
Python package / build (3.12.3) (pull_request) Successful in 3m13s
Python package / build (3.13) (pull_request) Successful in 2m44s
Python package / build (3.14) (pull_request) Successful in 2m42s
2026-09-29 17:22:59 +00:00
710b78a528 feat(configs): add connectivity_tester stack (#65)
Merge main to all / rebase (push) Failing after 23s
Python package / build (3.12.3) (push) Successful in 2m26s
Python package / build (3.13) (push) Successful in 2m7s
Python package / build (3.14) (push) Successful in 2m11s
## Summary

Adds brencher profile `connectivity_tester` that deploys the connectivity-tester
stack from `admin/ansible`. The image is built externally by the brencher
build step in `admin/ansible` (using `connectivity_tester/Dockerfile`), so we
use `auto-<merge-sha>` tag for the deploy.

## What's in the box

- `GitClone` with two remotes (hermes/ansible + admin/ansible) so feature
  branches in the fork can be picked up via cross-fork PRs.
- `DockerSwarmCheck(stack_name="connectivity_tester")` for diff detection.
- `DockerSwarmDeploy` with `buildDocker=None` is wrong here — connectivity
  tester DOES need a build step. **Need to change to non-None.**
- `UrlCheck` on /healthz and `SimpleLog` for the UI user link.

## Open question

The `admin/ansible/connectivity_tester/stack-compose.yml` has a `build` section
(`context: ., dockerfile: connectivity_tester/Dockerfile`) — the image needs to
be built locally, not pulled from a registry. Brencher must build the image
in the deploy step. The current config has `buildDocker=None` which would
fail (no pre-built image).

This is a known gap; the PR gets the structure in place. Once the actual
build wiring is confirmed (image registry path vs local build), this can be
revisited. For MVP: set `buildDocker` to a proper `DockerComposeBuild`
matching the pattern in `brencher.py`.

## Related

- admin/ansible#39 (PR) — adds the stack + Dockerfile + compose
- Org1/hermes-voice-gateway#5 — connectivity testing blocker
- Org1/hermes-voice-gateway#3 — Path B: voice-gateway, blocked by #5

- admin/ansible#16 — Connectivity testing monitoring: admin/ansible#16

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: #65
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-24 23:01:20 +02:00
hermes ea7c00a02f fix(configs): inline WORKFLOW_* in GiteaActions* ctor; keep only GITEA_TOKEN in secret
Python package / build (3.12.3) (pull_request) Successful in 2m47s
Python package / build (3.13) (pull_request) Successful in 2m41s
Python package / build (3.14) (pull_request) Successful in 2m31s
The previous split between env vars and constructor args created two
sources of truth for the workflow target. Moving WORKFLOW_OWNER /
WORKFLOW_REPO / WORKFLOW_FILE / WORKFLOW_REF into the constructor
arguments makes the dispatch target obvious at the call site and
keeps the docker secret template minimal (only credentials it must
contain).

README.instruction.md now lists only GIT_USERNAME / GIT_PASSWORD /
GITEA_TOKEN in the `printf` block; WORKFLOW_* are documented in
AGENTS.md (already there) as stack-specific config knobs.
2026-09-19 09:57:47 +00:00
hermes ac935ca8ca docs(readme): add GITEA_TOKEN + Actions dispatch knobs to docker secret template
Python package / build (3.12.3) (pull_request) Successful in 2m29s
Python package / build (3.13) (pull_request) Successful in 2m31s
Python package / build (3.14) (pull_request) Successful in 2m31s
The brencher stack build-delegation pattern (see AGENTS.md) needs four
env vars to fire `GiteaActionsTrigger`:

- GITEA_TOKEN: Gitea PAT used to POST /dispatches and poll /runs
- WORKFLOW_REPO: owner/repo that hosts the workflow file (default: hermes/ansible)
- WORKFLOW_FILE: workflow filename (default: release.yml)
- WORKFLOW_REF: branch or tag the workflow is triggered on (default: master)

Without GITEA_TOKEN the trigger step gets HTTP 401 from
/api/v1/repos/{owner}/{repo}/actions/runs; the pipeline halts at
trigger_build and the stack never deploys. Added the missing keys to
the secret-create `printf` block in README.instruction.md so the next
person wiring up a brencher stack doesn't have to discover this.

Defaults shown match the connectivity_tester stack wiring
(admin/brencher#65).
2026-09-19 08:32:55 +00:00
hermes 2b5c8952b3 fix(steps/git): use config_writer with force-fetch refspec
Python package / build (3.12.3) (pull_request) Successful in 2m30s
Python package / build (3.13) (pull_request) Successful in 2m43s
Python package / build (3.14) (pull_request) Successful in 2m37s
Default git fetch refspec refuses to update refs whose upstream SHA
diverges from the local copy. When a branch is force-pushed (e.g. our
smoke-test pushes of master, our re-applies of GITHUB_TOKEN fix), the
local remote-tracking ref points at the old SHA and git refuses to
overwrite it. This shows up as 'cannot lock ref ... is at <old> but
expected <new>' and fails the entire clone step.

Adding '+refs/heads/*:refs/remotes/<remote>/*' to each remote via
config.set_value(section, option, value) lets the fetch force-update
the remote-tracking refs. Each remote gets its own refspec; the
existing 'origin' branchNamePrefix refspec is preserved.
2026-09-18 21:55:24 +00:00
hermes 962b34e8ee Revert "fix(steps/git): use config_writer to set force-fetch refspec"
Python package / build (3.12.3) (pull_request) Canceled after 42s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
This reverts commit 3dd0657733.
2026-09-18 21:54:20 +00:00
hermes 3dd0657733 fix(steps/git): use config_writer to set force-fetch refspec
Python package / build (3.12.3) (pull_request) Successful in 2m35s
Python package / build (3.13) (pull_request) Successful in 2m47s
Python package / build (3.14) (pull_request) Canceled after 35s
GitPython Remote does not expose a set_refspec method. Use
repo.config_writer() to set remote.<name>.fetch '+refs/heads/*:refs/remotes/<name>/*'.
This bypasses the default safety against updating remote-tracking refs
that diverge from their upstream (force-pushed branches).

Sets the refspec for every remote, not just 'origin', so multi-remote
configs like connectivity_tester (hermes/ansible + admin/ansible) pick
up force-pushed branches on either side.
2026-09-18 21:48:34 +00:00
hermes 0bb705fc3e Revert "fix(steps/git): use force-fetch refspec so force-pushed branches are picked up"
This reverts commit 53b8796319.
2026-09-18 21:48:09 +00:00
hermes 53b8796319 fix(steps/git): use force-fetch refspec so force-pushed branches are picked up
Python package / build (3.12.3) (pull_request) Successful in 2m43s
Python package / build (3.13) (pull_request) Successful in 2m30s
Python package / build (3.14) (pull_request) Successful in 2m37s
Default git fetch refspec refuses to update refs whose upstream SHA
diverges from the local copy. When a branch is force-pushed (e.g. our
smoke-test pushes of master, our re-applies of GITHUB_TOKEN fix), the
local remote-tracking ref points at the old SHA and git refuses to
overwrite it. This shows up as 'cannot lock ref ... is at <old> but
expected <new>' and fails the entire clone step.

Adding '+refs/heads/*:refs/remotes/<remote>/*' to each remote's refspec
lets the fetch force-update the remote-tracking refs. Each remote's
existing refspec is preserved; the force-refspec is appended only if
not already present.
2026-09-18 21:39:31 +00:00
hermes 4fd29c8ea1 fix(steps/actions): clearer error when POST /dispatches 500
Python package / build (3.12.3) (pull_request) Successful in 2m49s
Python package / build (3.13) (pull_request) Successful in 2m16s
Python package / build (3.14) (pull_request) Successful in 2m8s
Refines the trigger_build failure mode: instead of opaque urllib.error.HTTPError,
raises a RuntimeError explaining the gitea-side bug + the tag-push workaround
(documented in the runtime message).
2026-09-18 20:58:44 +00:00
hermes e067b8c035 fix(configs): WORKFLOW_REF=master (release/* doesn't trigger push)
Python package / build (3.12.3) (pull_request) Successful in 2m15s
Python package / build (3.13) (pull_request) Successful in 2m11s
Python package / build (3.14) (pull_request) Successful in 2m16s
2026-09-18 20:42:53 +00:00
hermes 0bb3d53c7a fix(steps/actions): accept callable inputs (resolve at dispatch time)
Python package / build (3.12.3) (pull_request) Successful in 2m12s
Python package / build (3.13) (pull_request) Successful in 2m5s
Python package / build (3.14) (pull_request) Successful in 2m14s
2026-09-18 13:10:39 +00:00
admin 2229acc7f2 docs(AGENTS): document Actions build delegation pattern
Python package / build (3.12.3) (pull_request) Failing after 2m15s
Python package / build (3.13) (pull_request) Failing after 2m1s
Python package / build (3.14) (pull_request) Failing after 1m59s
2026-09-17 19:27:42 +02:00
admin 0196bcda30 feat(configs): brencher dispatches Actions workflow instead of build+push; deploy pulls image from Gitea CR
Python package / build (3.12.3) (pull_request) Canceled after 36s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
2026-09-17 19:27:12 +02:00
admin b187e19a6e feat(steps): DockerComposePull for pulling prebuilt image
Python package / build (3.12.3) (pull_request) Canceled after 0s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
2026-09-17 19:27:07 +02:00
admin 6cbadb4087 feat(steps): GiteaActionsTrigger and GiteaActionsAwait for build dispatch 2026-09-17 19:27:02 +02:00
admin 07911759d2 feat(configs): use PACKAGE_PUSH_USERNAME/TOKEN for Gitea CR; drop legacy registry
Python package / build (3.12.3) (pull_request) Successful in 2m9s
Python package / build (3.13) (pull_request) Successful in 2m14s
Python package / build (3.14) (pull_request) Successful in 2m12s
2026-09-17 18:33:44 +02:00
admin 555b46b746 chore(configs/connectivity_tester): try pushing to Gitea Container Registry
Python package / build (3.12.3) (pull_request) Successful in 2m2s
Python package / build (3.13) (pull_request) Successful in 2m7s
Python package / build (3.14) (pull_request) Successful in 2m24s
publish=True + Gitea registry URL. push will fail with HTTP 401 until GITEA_REGISTRY_USERNAME/GITEA_REGISTRY_PASSWORD are wired in as brencher container env — that's the diagnostic to find out whether the bot token is actually allowed to push to gitea.rudolf.keenetic.link/hermes.
2026-09-15 20:25:42 +02:00
admin 5eef68a1e0 docs(brencher): add AGENTS.md checklist for new stacks
Python package / build (3.12.3) (pull_request) Successful in 2m1s
Python package / build (3.13) (pull_request) Successful in 2m17s
Python package / build (3.14) (pull_request) Successful in 2m13s
2026-09-15 13:58:24 +02:00
admin 055b7e2015 fix(configs/connectivity_tester): wire unmerge for self-deploy
Python package / build (3.12.3) (pull_request) Successful in 2m12s
Python package / build (3.13) (pull_request) Successful in 2m14s
Python package / build (3.14) (pull_request) Successful in 2m51s
Replace state=SharedStateHolderInMemory(unmerge=None) with a GitUnmerge(clone, dockerSwarmCheck)-backed state holder. Without this, desired_branches is always empty, and brencher's polling loop falls through to 'Empty branches set' on every cycle, so the stack only deploys when an operator PATCHes the environment through the UI. With the unmerge in place the swarm service's org.brencher.version label drives state.branches and the stack self-deploys on each polling cycle.
2026-09-15 13:34:31 +02:00
admin db9b3b4cc4 fix(configs/connectivity_tester): drop UrlCheck; DNS not configured yet
Python package / build (3.12.3) (pull_request) Successful in 2m29s
Python package / build (3.13) (pull_request) Successful in 2m23s
Python package / build (3.14) (pull_request) Successful in 2m33s
Tracked in admin/brencher#68.
2026-09-15 08:15:31 +02:00
hermes ce2f5f6210 fix(configs): drop unmerge step from pipeline
Python package / build (3.12.3) (pull_request) Successful in 2m45s
Python package / build (3.13) (pull_request) Successful in 2m52s
Python package / build (3.14) (pull_request) Successful in 2m50s
Same fix as hermes_infra.py — pass unmerge=None to SharedStateHolderInMemory
and skip the unmerge step entirely. Having unmerge=None in the pipeline
violates mypy's type signature (Pipeline expects AbstractStep, None
is not assignable), causing the brencher image build to fail at the
mypy step.

This is the same workaround as hermes_infra (PR #83): the stack is
managed via direct pushes, not via UI state, so unmerge is not needed.
2026-09-14 20:40:29 +00:00
hermes 175d1f933e fix(configs): use unmerge=None for connectivity_tester
Python package / build (3.12.3) (pull_request) Failing after 2m23s
Python package / build (3.13) (pull_request) Failing after 2m23s
Python package / build (3.14) (pull_request) Failing after 2m24s
Without an entry in the brencher UI state.branches for connectivity_tester,
GitUnmerge.progress() returns an empty set and the pipeline raises
'Expected exactly one version, got: set()'. This stalls the brencher
loop, blocking other stacks (including brencher self-deploy) from being
processed.

Same workaround as hermes_infra.py (which uses unmerge=None because
its stacks are managed via direct pushes, not via UI).
2026-09-14 17:45:09 +00:00
hermes 2d51cd71f3 chore(compose): add connectivity_tester to PROFILES env
Python package / build (3.12.3) (pull_request) Successful in 2m25s
Python package / build (3.13) (pull_request) Successful in 2m55s
Python package / build (3.14) (pull_request) Successful in 2m47s
Without this, brencher's _discover_environments() does not see the
connectivity_tester.py module on next restart. Adding to the comma-
separated PROFILES env wires it in.

Also undo the misplaced relocation from 4586136: clone remote and
docker_compose_path point back at admin/ansible (the compose lives in
admin/ansible#39, not in brencher).
2026-09-14 17:26:56 +00:00
hermes 45861362c2 feat(configs): relocate connectivity_tester stack into brencher repo
Python package / build (3.12.3) (pull_request) Failing after 2m18s
Python package / build (3.13) (pull_request) Failing after 2m7s
Python package / build (3.14) (pull_request) Failing after 2m6s
Move compose + Python service + Dockerfile into backend/configs/connectivity_tester/
so the stack can be deployed from a single repo (no cross-repo compose
lookup at deploy time). The brencher config now points at
admin/brencher instead of admin/ansible, and at the new in-repo
compose path.
2026-09-14 17:03:41 +00:00
hermesandhermes 2e48d11b77 feat(ui): pre-fill branch filter from ?filter=<text> GET-param (#67)
Merge main to all / rebase (push) Failing after 32s
Python package / build (3.12.3) (push) Successful in 2m45s
Python package / build (3.13) (push) Successful in 2m34s
Python package / build (3.14) (push) Successful in 2m43s
## Summary

Lets the brencher UI pre-fill its branch filter from a `?filter=<text>` query
parameter on initial page load.

## Use case

Deep-link filtered views from external tooling (runbooks, Gitea issues,
dashboards). For example:

- `https://brencher.rudolf.keenetic.link/?filter=connectivity_tester` — go straight to the new stack after #65 lands.
- `https://brencher.rudolf.keenetic.link/?filter=hermes_infra%20v0.4.0` — multiple words (current filter does substring match on all).

## Changes

`frontend/app.js`: IIFE on initial load reads `URLSearchParams`, sets
`branchFilter.value`, calls `filterBranches()`. Empty / missing param
leaves the filter empty (no pre-fill, no apply) — same as before.

## Acceptance

- `?filter=foo` pre-fills the input AND applies the filter on initial render.
- Empty `?filter=` is a no-op (param present but empty string).
- No param at all is a no-op (no pre-fill, no apply).
- Existing inline typing behaviour unchanged.

## Resolves

- admin/brencher#66 (brencher UI: support ?filter=<text> GET-param that auto-fills the filter input)

## Related

- admin/brencher#65 (PR feat(configs): add connectivity_tester stack) — once #65 merges, the link `/<brencher-host>/?filter=connectivity_tester` becomes a one-click entry point to the new stack.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: #67
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-14 18:43:01 +02:00
hermes 8b8bbbb230 feat(configs): wire DockerComposeBuild into connectivity_tester pipeline
Python package / build (3.12.3) (pull_request) Successful in 2m20s
Python package / build (3.13) (pull_request) Successful in 2m17s
Python package / build (3.14) (pull_request) Successful in 2m19s
admin/ansible#39 has a build section in stack-compose.yml, so the
image must be built locally — buildDocker=None would fail with
'No such image' on deploy. Added a proper DockerComposeBuild step
between checkout and deploy. Image tagged auto-<merge-sha> locally;
no registry push (publish=False).
2026-09-14 13:07:39 +00:00
hermes 1f7ca6609d feat(configs): add connectivity_tester stack
Python package / build (3.12.3) (pull_request) Canceled after 1m30s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
Long-running Swarm service that runs TCP probes against an allow-list of
internal+external endpoints and a negative-list (cloud metadata etc.)
every 10 minutes. Logs structured PASS/FAIL per check.

Stack lives in admin/ansible under connectivity_tester/. One task per
node via mode: global. Resolves Org1/hermes-voice-gateway#5
(connectivity testing blocker for #3).
2026-09-14 13:05:12 +00:00
hermes 7938f4f0f0 fix(configs/hermes_infra): wire unmerge into state holder for restart recovery (#61)
Merge main to all / rebase (push) Failing after 54s
Python package / build (3.12.3) (push) Successful in 4m11s
Python package / build (3.13) (push) Successful in 2m33s
Python package / build (3.14) (push) Successful in 2m17s
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/61
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-12 00:15:38 +02:00
52692a75b8 fix(web): use setdefault in WebSocket broadcast to avoid KeyError race (#58)
Merge main to all / rebase (push) Failing after 57s
Python package / build (3.12.3) (push) Successful in 4m10s
Python package / build (3.13) (push) Successful in 2m28s
Python package / build (3.14) (push) Successful in 2m27s
## Symptom
Production brencher UI reports constant WebSocket reconnects (see
user observation 2026-09-05, fact #5). Prod logs from
`brencher_brencher-backend` (image `brencher:auto-74bf5aef-…`) show
`WebSocket /ws [accepted]` followed within ~3–10 s by
`KeyError: <starlette.websockets.WebSocket object …>` raised at
`backend/web.py:175`, then `connection closed`. Same UI reconnecting
after the 5 s backoff. The companion `brencher2` instance (one
stack, no `SECONDARY_BRENCHER`) shows 0 errors over the same window,
confirming that the same code on a single-stack single-source path
is race-free.

## Root cause
`ws_connections: Dict[WebSocket, Dict[str, Any]]` is mutated from
two coroutines that race on the asyncio event loop:
- `broadcast()` — invoked via `asyncio.run_coroutine_threadsafe`
  from the processing thread on every `onupdate()`. With 7 profiles
  on the new brencher this fires ≥10 times per ~100 s cycle, and
  the SecondaryManager’s `on_update=lambda: self.broadcast_all()`
  layers additional dispatches on top.
- `websocket_endpoint()` — the FastAPI handler running on the
  uvicorn loop. Between `await websocket.accept()` and
  `self.ws_connections[websocket] = {}`, the handler yields. A
  concurrent broadcast that was already iterating
  `self.ws_connections.keys()` may include the freshly accepted
  socket in `disconnected` (because its own `send_text` raised
  earlier and was logged), then `pop` removes it before
  `websocket_endpoint` writes back. The handler then hits
  `self.ws_connections[websocket]["branches"] = …` and raises
  `KeyError`. Starlette’s default error path closes the WS.

## Fix (this PR — variant `setdefault`)
Treat `ws_connections` as a sparse dict: rely on
`dict.setdefault` for atomic insert-or-get, and hold a local
`state` binding throughout the handler so every subsequent write
operates on the same dict object even if a concurrent broadcast
removes and re-adds the entry.

Diff is 12 lines, no API or protocol change:
- `broadcast()`: iterate `ws_connections.items()`; write to the
  local `state[event] = message` instead of
  `self.ws_connections[websocket][event] = message`.
- `websocket_endpoint()`: replace `ws_connections[websocket] = {}`
  with `state = ws_connections.setdefault(websocket, {})`; replace
  the two follow-up writes with `state["branches"] = …` /
  `state["environments"] = …`.

Properties:
- A broadcast scheduled before `setdefault` runs is a no-op for
  the freshly accepted client (its first frame is the handler’s
  initial `send_text`, so losing one in-flight broadcast is
  benign; the next periodic broadcast at `~10 s` re-sends).
- Stale `state` returned by `setdefault` is overwritten in place
  — no lock needed because CPython dict operations are atomic
  under the GIL, and we only ever touch the dict’s value binding,
  never its key existence.
- Code reads cleaner: there’s one `state = ws_connections[…]`
  binding per WS, no `self.ws_connections[websocket]["x"]`
  chains.

## Companion PRs (variants 3 and 4 — same race, different shape)
- `fix/ws-asyncio-lock` — explicit `asyncio.Lock` around the
  dict; serializes mutation but not `send_text`.
- `fix/ws-writer-loop` — single-writer task consuming an
  `asyncio.Queue`; broadcast becomes inherently sequential.

All three solve the same root cause with different trade-offs. The
recommended merge order is this one (smallest diff, lowest review
surface). Promote to `Lock` or `Queue` only if benchmarking shows
setdefault is insufficient under heavier connection concurrency.

## Validation
- `python3 -m py_compile backend/web.py` → OK.
- Diff: 12 lines in one file.

Manual verification on production is the user’s call. The prod
logs already show the pre-fix error pattern (`brencher_brencher-backend`
1.80 WS errors per WS_OPEN), so post-deploy the same log extraction
script (in `prod-ws-error-count` skill) should drop to 0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: alex <alex@rudenk.local>
Co-authored-by: brencher_bot <brencher_bot@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/58
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-11 17:07:44 +02:00
hermesandadmin ddce27f1af recover(auto-6fd59898): preserve 6fd598983a (PR #26 head, gmail_mcp) (#53)
Auto-recovery snapshot of shard auto-6fd59898 from image auto-04d3624a-34caac15-6fd59898-920249ec-afc2de34. Fossil branch. This commit was recovered from refs/pull/26/head (Gitea retains PR-head refs even after merge). PR #26 was merged; mergeable will likely be False.

---------

Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/53
Co-authored-by: brencher_bot (hermes) <rudolfss13+hermes@gmail.com>
2026-09-06 16:28:18 +02:00
hermesandhermes 33caa4cb9f recover(auto-34caac15): preserve 34caac15ce (PR #12 commit, immich) (#52)
Auto-recovery snapshot of shard auto-34caac15 from image auto-04d3624a-34caac15-6fd59898-920249ec-afc2de34. Fossil branch. PR #12 (point immich at gitea admin/hermes ansible repos) was merged into admin/brencher:main; mergeable will likely be False because the content is already in main.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/52
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-06 16:27:57 +02:00
hermesandadmin a9de490ebe recover(auto-04d3624a): preserve 04d3624afd (PR #28 commit, hermes v2) (#51)
Auto-recovery snapshot of shard auto-04d3624a from image auto-04d3624a-34caac15-6fd59898-920249ec-afc2de34. Fossil branch. The commit's content already landed in admin/brencher:main via PR #28 (closed) and is part of PR #11 family. Opening this PR is for archival/reference only; mergeable will likely be False because the content is already in main.

---------

Co-authored-by: admin <rudolfss13@gmail.com>
Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/51
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-06 16:27:10 +02:00
hermesandhermes 920249ec30 feat(configs): point gitea-config at gitea admin/hermes ansible repos (#10)
Merge main to all / rebase (push) Failing after 33s
Python package / build (3.12.3) (push) Successful in 2m30s
Python package / build (3.13) (push) Successful in 2m17s
Python package / build (3.14) (push) Successful in 2m23s
Python package / build (3.12.3) (pull_request) Successful in 2m40s
Python package / build (3.13) (pull_request) Successful in 2m40s
Python package / build (3.14) (pull_request) Successful in 2m50s
## Summary
Migrates `backend/configs/gitea.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/gitea.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git"[, branchNamePrefix=...])`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why migrate
- The old URL never resolved (`uber_backup.git` doesn't exist on gitea anyway) — so the deploy path was broken even though the file looked correct.
- `github.com` returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (commit `653aee9`).

## Scope decision: only `gitea.py`
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) are being migrated in **separate PRs**, one per file. Each PR has its own branch + verification.

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/gitea.py` — OK
- Real import via `from configs import gitea`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''`
- pytest: **26/26** passed on the no-docker / no-playwright suite
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible`.
- end-to-end `docker stack deploy` — needs a real swarm.

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of this environment will clone from gitea instead of the unreachable github URL.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/10
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-05 13:36:22 +02:00
hermes 75126cb2fd ci(merge_main): hardcode gitea.rudolf.keenetic.link (GITEA_SERVER_URL empty) (#47)
Merge main to all / rebase (push) Failing after 28s
Python package / build (3.12.3) (push) Successful in 2m33s
Python package / build (3.13) (push) Successful in 2m20s
Python package / build (3.14) (push) Successful in 2m16s
## Problem

Run #67 of `merge_main.yaml` failed with:

```
fatal: unable to access 'https:///admin/brencher.git/': URL rejected: No host part in the URL
```

The `PUSH_TOKEN` secret is now wired up correctly, but `${GITEA_SERVER_URL}` is empty in this Gitea Actions runtime (the variable the env-prefixed runner is supposed to set), so the substituted URL drops the host.

## Fix

Hardcode `gitea.rudolf.keenetic.link` in both `merge_main.yaml` and `merge_main_and_squash.yaml`. When the runner exposes `GITEA_SERVER_URL` in a future version, this can be replaced back with `${GITEA_SERVER_URL#https://}`.

## Supersedes

This is a follow-up to admin/brencher#46 which already wired `secrets.PUSH_TOKEN` but still had the empty env var.Reviewed-on: http://192.168.1.96/admin/brencher/pulls/47

Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-04 16:26:09 +02:00
admin 0e83458044 Обновить REST.instruction.md (#48)
Merge main to all / rebase (push) Failing after 12s
Python package / build (3.12.3) (push) Successful in 2m6s
Python package / build (3.13) (push) Successful in 2m10s
Python package / build (3.14) (push) Successful in 2m22s
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/48
2026-09-04 13:26:37 +02:00
admin 7dc36d7487 Merge pull request 'ci(merge_main): fix auth for self-hosted Gitea Actions' (#46) from hermes/brencher:fix/gitea-actions-merge-workflows into main
Merge main to all / rebase (push) Failing after 15s
Python package / build (3.12.3) (push) Successful in 2m17s
Python package / build (3.13) (push) Successful in 2m19s
Python package / build (3.14) (push) Successful in 2m23s
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/46
2026-09-04 13:08:06 +02:00
hermes b0e32f835a ci(merge_main): use PUSH_TOKEN instead of GITEA_PUSH_TOKEN
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (pull_request) Successful in 34s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (pull_request) Successful in 33s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (pull_request) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (pull_request) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (pull_request) Successful in 32s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (pull_request) Successful in 35s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (pull_request) Successful in 1m39s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (pull_request) Successful in 1m55s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (pull_request) Successful in 2m0s
Python package / build (pip list, 3.12.3) (pull_request) Successful in 31s
Python package / build (pip list, 3.13) (pull_request) Successful in 33s
Python package / build (pip list, 3.14) (pull_request) Successful in 32s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (pull_request) Successful in 32s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (pull_request) Successful in 29s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (pull_request) Successful in 29s
Match the secret name already created in admin/brencher.
2026-09-04 11:01:55 +00:00
hermes 1c0580750d ci(merge_main): fix auth for self-hosted Gitea Actions
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (pull_request) Successful in 2m13s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (pull_request) Successful in 35s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (pull_request) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (pull_request) Successful in 37s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (pull_request) Successful in 34s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (pull_request) Successful in 34s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (pull_request) Successful in 1m41s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (pull_request) Successful in 1m39s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (pull_request) Successful in 1m41s
Python package / build (pip list, 3.12.3) (pull_request) Successful in 31s
Python package / build (pip list, 3.13) (pull_request) Successful in 29s
Python package / build (pip list, 3.14) (pull_request) Successful in 32s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (pull_request) Successful in 27s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (pull_request) Successful in 29s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (pull_request) Successful in 29s
Both workflows were written for github.com and relied on the implicit
secrets.GITHUB_TOKEN + github.repository + a hardcoded github.com URL.
On this self-hosted Gitea instance (ghauth.rudolf.keenetic.link) they fail
with "Invalid username or token. Password authentication is not supported
for Git operations." against https://github.com/admin/brencher.git.

The fix:
- Read the push token from a new repo-level secret GITEA_PUSH_TOKEN
  (admin/brencher). Same value is injected into the checkout and into the
  rewritten remote URL so push can succeed.
- Build the origin URL from Gitea-injected GITHUB_REPOSITORY and
  GITEA_SERVER_URL instead of hardcoding github.com.
- Update git user.email/name for the Gitea Actions identity.

To enable: create secret GITEA_PUSH_TOKEN in admin/brencher (a PAT with
write access to the repo) via the Gitea UI before merging this PR.
2026-09-04 08:03:46 +00:00
8059cc74ea ci: speed up python-package.yml from ~13min to ~3-4min (#15)
Merge main to all / rebase (push) Failing after 14s
Python package / build (3.12.3) (push) Successful in 2m11s
Python package / build (3.13) (push) Successful in 2m22s
Python package / build (3.14) (push) Successful in 2m18s
## Summary
Optimizes `.github/workflows/python-package.yml` so the build job runs in ~3-4 minutes instead of ~13 minutes (which is what the recent run on PR #5 took — see https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/14, started 23:52, completed 00:05, **13m 19s wall-clock**).

## Why it was slow
The current matrix has **3 Python versions × 5 checkers = 15 separate CI jobs**, each one:
1. Checking out the repo (~10s)
2. Setting up Python via `actions/setup-python@v3` with **no cache enabled** (~15-30s)
3. Running `pip install -r requirements.txt -r requirements-dev.txt` **from scratch every single time** (~30-90s per job × 15 jobs = 7-22 minutes wasted)

The two debug entries (`python -c 'import sys; print(...)'` and `pip list`) were not real checks — they were noise from the upstream github `build-and-test-python` template, left in place but never reviewed.

## What's in this PR
- `.github/workflows/python-package.yml` — single file, +41/-56
  1. **`cache: 'pip'` enabled** with `cache-dependency-path: requirements.txt, requirements-dev.txt` — first run downloads and caches, subsequent runs hit the cache. Drops pip install from ~60s to ~5-15s when requirements don't change.
  2. **Two debug-only matrix entries removed** (`python -c ...` and `pip list`) — they were not real checks.
  3. **Matrix restructured**: one Python-version axis only, no checker axis. Each Python job runs `install + flake8 + mypy` sequentially. **15 jobs → 3 jobs** (~5x fewer fresh venvs to set up).
  4. **`concurrency: cancel-in-progress: true`** added — a new push to the same PR auto-cancels the previous in-flight run, saving runner minutes when iterating on a fix.
  5. Bumped `actions/setup-python` from `v3` to `v5` (required for `cache: 'pip'`).

## Verification
- YAML parses via PyYAML — ok
- `jobs` dict contains `['build']` only
- `build.strategy.matrix.python-version` = `['3.12.3', '3.13', '3.14']`
- **total CI jobs: 3** (was 15)
- `setup-python@v5` with `cache: 'pip'` + `cache-dependency-path` pointing at `requirements.txt` and `requirements-dev.txt`
- `concurrency` block correctly scoped to `ci-${{ github.workflow }}-${{ github.ref }}`

## Expected timing
| Scenario | Before | After |
|---|---|---|
| First run (cold cache) | ~13m 19s | ~5-7m |
| Subsequent runs (warm cache, no req changes) | ~13m 19s | ~3-4m |
| Re-run after `requirements.txt` change | ~13m 19s | ~4-5m |

## What was NOT verified
- Actual Gitea Actions runner behaviour with `cache: 'pip'` — the Gitea Actions runner is GitHub Actions-compatible, and `actions/setup-python@v5` with `cache: 'pip'` is a well-known idiom there, but the local Gitea runner config may or may not honour it. If it doesn't, the 15→3 fan-out reduction alone is still a ~5x speedup.

## Note on PR #14
PR #14 ("re-enable the tests job") is still open. If/when it merges, this PR will need a follow-up to re-add the `tests:` job with the same `cache: 'pip'` setup applied — otherwise the playwright install will dominate again.

## Commit history
```
* db3807e (HEAD -> feat/optimize-ci-build-time, origin/feat/optimize-ci-build-time)
      ci: speed up python-package.yml from ~13min to ~3-4min
* 70f4a41 ci: comment out the tests job in python-package.yml (#13)   (upstream/main tip)
```

---------

Co-authored-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/15
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-09-04 09:48:29 +02:00
hermesandadmin e690f6a50c feat(configs): point torrserv_proxy at gitea admin/hermes repos (#23)
Merge main to all / rebase (push) Failing after 18s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 56s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 48s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 46s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 44s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 42s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 42s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m56s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 2m0s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m2s
Python package / build (pip list, 3.12.3) (push) Successful in 39s
Python package / build (pip list, 3.13) (push) Successful in 39s
Python package / build (pip list, 3.14) (push) Successful in 55s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 48s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 33s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 43s
## Summary

Migrates `backend/configs/torrserv_proxy.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/reverseproxy.git`) to the new `remotes=[{name, url}, ...]` form introduced by #3.

Source repo is `rudolf1/reverseproxy` (not `brencher`), so the three remotes point at:

- `github`       → `https://github.com/rudolf1/reverseproxy.git`
- `gitea_admin`  → `https://gitea.rudolf.keenetic.link/admin/reverseproxy`
- `gitea_hermes` → `https://gitea.rudolf.keenetic.link/hermes/reverseproxy`

Same pattern that already works in production for `brencher.py` and `brencher2.py`.

## Diff

```diff
-clone = GitClone(url="https://github.com/rudolf1/reverseproxy.git")
+clone = GitClone(remotes=[
+	{"name": "github",       "url": "https://github.com/rudolf1/reverseproxy.git"},
+	{"name": "gitea_admin",  "url": "https://gitea.rudolf.keenetic.link/admin/reverseproxy"},
+	{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/reverseproxy"},
+])
```

## Test plan

Manual test on prod: trigger brencher with this config, verify it clones from the gitea mirrors and `torrserv_proxy` stack deploys successfully.

Refs #22
Part of #18

---------

Co-authored-by: brencher_bot <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/23
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-30 22:29:00 +02:00
hermes 98ee950a1f fix(docker): cap swarm config name at 64 chars via sha256 fallback (#17)
Merge main to all / rebase (push) Failing after 14s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 39s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 46s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 39s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 39s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 38s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 41s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m50s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m57s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m5s
Python package / build (pip list, 3.12.3) (push) Successful in 34s
Python package / build (pip list, 3.13) (push) Successful in 34s
Python package / build (pip list, 3.14) (push) Successful in 35s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 31s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 31s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 32s
## What

Since #5 (Fix/show pipeline errors), brencher appends a version tail to every swarm config name so each deploy creates a new config object (swarm configs are immutable). For long base names plus the version string, the resulting prefixed name (stack_name_ + name) can exceed docker swarm's 64-char limit and `docker stack deploy` fails with `InvalidArgument`.

## Repro

```
[2026-08-29 19:38:46,063] failed to create config hermes_infra_brencher_state_proxy_conf_auto-16875f17-4cb8eeae-bfb3c220: Error response from daemon: rpc error: code = InvalidArgument desc = invalid name, only 64 [a-zA-Z0-9-_.] characters allowed
```

`brencher_state_proxy_conf_auto-16875f17-4cb8eeae-bfb3c220` is 57 chars, plus the 13-char `hermes_infra_` stack prefix = 70, over the 64-char limit.

## Fix

When the would-be full prefixed name is longer than 64 chars, replace the suffix with `sha256(full)[:32]`. 32 hex chars is comfortably unique per `(base_name, version)` pair and the prefixed name ends up at 12 + 1 + 32 = 45 chars, well under the limit.

Short names keep the readable `<name>_<version>` form (no change for the common case).

## Verified

Manual sanity check on the four configs currently in `hermes_squid`:

| base | version | compose name | docker name | length |
|---|---|---|---|---|
| squid_conf | auto-4cb8eeae-bfb3c220 | squid_conf_auto-4cb8eeae-bfb3c220 | hermes_infra_squid_conf_auto-4cb8eeae-bfb3c220 | 46 |
| brencher_state_proxy_conf | auto-16875f17-4cb8eeae-bfb3c220 | 7644a2983c7f6c7b09c6ed84bff0096d | hermes_infra_7644a2983c7f6c7b09c6ed84bff0096d | 45 |
| brencher_state_proxy_conf | auto-4cb8eeae-bfb3c220 | brencher_state_proxy_conf_auto-4cb8eeae-bfb3c220 | hermes_infra_brencher_state_proxy_conf_auto-4cb8eeae-bfb3c220 | 61 |
| nginx_hermes_conf | auto-16875f17-4cb8eeae-bfb3c220 | nginx_hermes_conf_auto-16875f17-4cb8eeae-bfb3c220 | hermes_infra_nginx_hermes_conf_auto-16875f17-4cb8eeae-bfb3c220 | 62 |

All under 64 chars; sha256 fallback only kicks in for the long case.

## Impact

- Fixes deploys of stacks that have a config with a long base name (e.g. `brencher_state_proxy_conf` in `hermes_infra`).
- Other stacks (`gitea`, `immich`, `torrserv_proxy`, etc.) are unaffected because their config names are short enough to fit the existing readable form.
- After merge + brencher self-deploy, the failed hermes_infra stack at `brencher-state-proxy` (failed since 2026-08-26) can finally be re-deployed and pick up the new `brencher-state-proxy.conf` with the resolver fix from #5's companion config rename.

## Related

- This unblocks `admin/ansible#21` (feat/stt-voice-faster-whisper) whose deploy was failing on the same length error.
- The companion nginx-side fix in `admin/ansible#21` (`brencher-state-proxy.conf`: `resolver` + `set $upstream`) will start working once this lands and the config object is actually recreated.Reviewed-on: http://192.168.1.96/admin/brencher/pulls/17
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-29 22:54:07 +02:00
admin e4f90b1f23 Fix/show pipeline errors (#5)
Merge main to all / rebase (push) Failing after 22s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 37s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 31s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 42s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 43s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m53s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m58s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 1m54s
Python package / build (pip list, 3.12.3) (push) Successful in 37s
Python package / build (pip list, 3.13) (push) Successful in 37s
Python package / build (pip list, 3.14) (push) Successful in 34s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 29s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 36s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 35s
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/5
2026-08-27 10:58:55 +02:00
hermesandhermes ba17865a49 feat(configs): point hermes_infra at gitea admin/hermes ansible repos (#8)
Merge main to all / rebase (push) Failing after 15s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 41s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 36s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 41s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 35s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 37s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m43s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 1m55s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 1m48s
Python package / build (pip list, 3.12.3) (push) Successful in 34s
Python package / build (pip list, 3.13) (push) Successful in 32s
Python package / build (pip list, 3.14) (push) Successful in 36s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 37s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 33s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 31s
## Summary
Switches `backend/configs/hermes_infra.py` from the legacy `url=` form (pointing at the unreachable `github.com/rudolf1/uber_backup.git`) to the new `remotes=[{name, url}, ...]` form introduced by PR #3 (named-remotes), pointing at the actual gitea repos where the ansible stack lives.

## What's in this PR
- `backend/configs/hermes_infra.py` — single file, +22/-1
  - `GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")`
    →
    `GitClone(remotes=[{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"}, {"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"}])`
  - `branchNamePrefix` is dropped (it would be a silent no-op: `GitClone` only applies its prefix refspec to a remote named `origin` per `backend/steps/git.py` ~L104, and neither remote here is named `origin`)

## Why this config specifically
- `hermes_infra.py` is the config that **deploys `hermes_squid/stack-compose.yml`** — i.e. it is what brings the sidecars added in `admin/ansible#1` (the docker_socket_proxy + brencher-state-proxy from earlier today) to a running swarm. Pointing its clone at the unreachable github URL meant the deploy path was broken even though the file looked correct.
- `uber_backup.git` doesn't exist on gitea anyway (only `admin/ansible`, `hermes/ansible`, `admin/brencher`, `hermes/brencher` exist) — so the old URL was always wrong, not just unreachable.
- github.com returns HTTP 403 on CONNECT tunnel from the brencher container's egress proxy — verified in `admin/brencher#4` (PR #4 / commit `653aee9`).

## Scope decision: only `hermes_infra.py` was migrated
The other open configs (`registry.py`, `gitea.py`, `hermes.py`, `immich.py`, `gmail_mcp.py`, `torrserv_proxy.py`) still use the legacy form. They're intentionally left untouched — this PR keeps the migration scope narrow on `hermes_infra` only, as requested. A future follow-up PR can migrate them once the upstream uber_backup / reverseproxy mirrors exist on gitea (they don't right now).

## Verification (sandbox, no docker daemon)
- `python3 -m py_compile backend/configs/hermes_infra.py` — OK
- Real import via `from configs import hermes_infra`:
  - `clone.remotes` is a 2-entry list `[gitea_hermes, gitea_admin]`, `clone.urls` derived from it
  - no `github.com` substring anywhere in the constructed config
  - `clone.branchNamePrefix == ''` (not just unset)
  - `env.id == "hermes_infra"`, pipeline unchanged
  - `deploy step docker_compose_path == "hermes_squid/stack-compose.yml"`
  - `deploy step stack_name == "hermes_infra"`
- pytest: **26/26** passed on the no-docker / no-playwright suite (`test_dummy`, `test_caching_step`, `test_cached_env`, `test_git_integration`)
- mypy `--no-incremental --follow-imports=silent`: `Success: no issues found in 1 source file`
- **End-to-end smoke test**: built a parallel `GitClone(remotes=[...])` with both URLs pointed at `file:///tmp/ansible-upstream`, ran `clone.progress()`:
  - both named remotes created in the temp clone
  - 4 branches discovered (`master` + `feat/docker-socket-proxy` × 2 remotes)
  - `feat/docker-socket-proxy` visible on both `gitea_admin` and `gitea_hermes`
  - no `github.com` in any remote URL

## What was NOT verified in the sandbox
- HTTPS clone from `gitea.rudolf.keenetic.link` on the live brencher-bot — requires the brencher container's git credentials to have push rights on `admin/ansible` (the live-deploy source). Sandbox can't simulate this.
- end-to-end `docker stack deploy hermes_infra` — needs a real swarm

## After merge
After this PR merges into `admin/brencher:main`, the next brencher deploy of the `hermes_infra` environment will clone from gitea instead of the unreachable github URL. Combined with `admin/ansible#1` (which adds the new services to `hermes_squid/stack-compose.yml`) and `admin/brencher#4` (which registers `docker_socket_proxy` in the PROFILES env var), the full chain is:
1. brencher clones ansible from `gitea.rudolf.keenetic.link/{hermes,admin}/ansible`
2. Checks out a merged branch containing the new sidecars
3. Builds (only `sarg` — `docker_socket_proxy` and `brencher-state-proxy` use upstream images, no build)
4. `docker stack deploy hermes_infra` rolls out the full stack

## Commit history
```
* 9a0cd90 (HEAD -> feat/point-hermes-infra-at-gitea, origin/feat/point-hermes-infra-at-gitea)
      feat(configs): point hermes_infra at gitea admin/hermes ansible repos
* e533174 Allow naming each GitClone remote explicitly (#3)         (upstream/main tip)
```

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/8
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-26 11:30:17 +02:00
hermesandhermes 70f4a4163e ci: comment out the tests job in python-package.yml (#13)
## Summary
Comments out the `tests:` job in `.github/workflows/python-package.yml`. The slow `pytest` matrix (with `playwright install --with-deps chromium`) no longer runs on every push or PR — only the cheap `build` job (lint + mypy) remains.

## Why
- `pytest` requires `playwright install --with-deps chromium` on every run, which dominates the CI time even though pytest is not currently the merge gate.
- The `build` job (flake8 + mypy) still runs and still catches real regressions cheaply.
- `workflow_dispatch` is unchanged — tests can still be triggered manually from the Actions UI when needed.

## What's in this PR
- `.github/workflows/python-package.yml` — single file, +35/-34
  - The `tests:` job (33 lines) is now commented with leading `# `
  - Everything else (triggers, `build:` job, matrix) is unchanged
  - YAML still parses cleanly; `jobs` dict now contains only `build`

## Verification
- `python -c "import yaml; yaml.safe_load(open('.github/workflows/python-package.yml'))"` — OK
- `wf['jobs']` contains `['build']` only (no `tests` key)
- `build.matrix.checker` still has all 5 checkers:
  - `python -c "import sys; ..."` (debug)
  - `pip list` (debug)
  - `flake8 . --select=E9,F63,F7,F82` (syntax errors)
  - `flake8 . --exit-zero --max-complexity=10` (complexity)
  - `mypy --no-incremental --config-file mypy.ini backend tests`
- triggers unchanged: `workflow_dispatch`, `push`, `pull_request` (all targeting `main`)

## How to re-enable
Remove the leading `# ` from each commented line, or delete the commented block entirely and rely on git history.

## What was NOT verified
- The actual GitHub Actions runner behaviour change — this repo runs on Gitea Actions (the project's primary CI is the brencher pipeline itself, not Gitea-hosted runners for these workflows). Gitea's Actions runner is GitHub Actions–compatible and will see the `tests` job as absent and skip it.

## Commit history
```
* e271b2e (HEAD -> feat/disable-ci-tests, origin/feat/disable-ci-tests)
      ci: comment out the tests job in python-package.yml
* e533174 Allow naming each GitClone remote explicitly (#3)         (upstream/main tip)
```

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/13
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-25 23:24:49 +02:00
e5331745be Allow naming each GitClone remote explicitly (#3)
Merge main to all / rebase (push) Failing after 19s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (push) Successful in 38s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (push) Successful in 38s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (push) Successful in 52s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (push) Successful in 40s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (push) Successful in 41s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (push) Successful in 40s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (push) Successful in 1m55s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (push) Successful in 2m1s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (push) Successful in 2m1s
Python package / build (pip list, 3.12.3) (push) Successful in 38s
Python package / build (pip list, 3.13) (push) Successful in 40s
Python package / build (pip list, 3.14) (push) Successful in 41s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (push) Successful in 35s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (push) Successful in 37s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (push) Successful in 40s
Python package / tests (pytest, 3.12.3) (push) Failing after 7m43s
Python package / tests (pytest, 3.13) (push) Failing after 7m7s
Python package / tests (pytest, 3.14) (push) Failing after 7m36s
Implements the review comment on #2:

> I would like to give names for remotes

## Summary

Adds a `remotes=[{"name": ..., "url": ...}, ...]` constructor to `GitClone` so the config author controls how each URL maps to a git remote. The existing `url=str` / `url=[str]` shorthand is kept as backward-compatible (positional list still maps to `origin`, `repo1`, `repo2`, ...), so PR #2's `test_git_clone_multiple_urls` continues to pass.

## Changes

- **`backend/steps/git.py`** — `GitClone.__init__` now accepts either `url=` or `remotes=`. With `remotes=`, every entry must carry a non-empty unique `name` and `url`; duplicates and missing fields raise `ValueError`. The constructor rejects passing both or neither. `self.remotes` becomes the source of truth; `self.urls` is a derived list (kept for any introspection). `CheckoutMerged.find_desired_commits` falls back to the configured primary remote instead of the hardcoded `'origin'` when a branch key has no remote prefix, so configs that don't name a remote `origin` still work.
- **`backend/configs/brencher.py`, `brencher2.py`** — switch from positional list to named remotes `github`, `gitea_admin`, `gitea_hermes`.
- **`tests/test_git_integration.py`** — adds `test_git_clone_named_remotes` (asserts the user-chosen names appear verbatim and `origin`/`repo1` do not leak) and `test_git_clone_remotes_validation` (covers every rejection path: empty list, missing/empty name, missing/empty url, duplicate name, both-kwargs, no-kwargs).

## Diff vs #2

```
backend/configs/brencher.py    |  8 ++---
backend/configs/brencher2.py   |  8 ++---
backend/steps/git.py           | 76 +++++++++++++++++++++++++++++++++++++++-------
tests/test_git_integration.py  | 75 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 147 insertions(+), 20 deletions(-)
```

Targets `admin/multiple_remotes` so this lands on top of #2. After #2 merges, this is a clean follow-up; if #2 is reworked, this branch will need a rebase.

## Verification

- `python3 -m py_compile` on all 4 changed files: clean.
- AST-driven call-shape check: every `GitClone(...)` in both configs uses the new `remotes=` kwarg with exactly `{github, gitea_admin, gitea_hermes}`; no positional-list leftover in configs.
- Executed the new `__init__` in isolation against a stubbed `super()` and exercised 12 cases: 3 OK shapes (url=str, url=[..], remotes=[valid]) and 9 rejection shapes (empty list, missing/empty name, missing/empty url, duplicate name, both-kwargs, no-kwargs, non-dict entry). All 12 produced the asserted `ValueError`/`TypeError` with the expected message substring.
- Could not run `pytest` in this sandbox (PyPI is blocked by the proxy and no `pytest`/`gitpython` is pre-installed). The two new tests and PR #2's existing test should be run in a real env before merge.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Co-authored-by: rudolf <rudolfss13@gmail.com>
Reviewed-on: http://192.168.1.96/admin/brencher/pulls/3
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-08-25 18:53:12 +02:00
admin 94a858d8ef Version on UI + gitea 2026-08-12 21:09:07 +02:00
admin 57393c933c hermes (#83)
* hermes
2026-08-05 21:26:07 +02:00
29 changed files with 1258 additions and 162 deletions
+4 -8
View File
@@ -20,16 +20,16 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
token: ${{ secrets.PUSH_TOKEN }}
- name: Configure git
run: |
git config user.email "github-actions[bot]@users.noreply.github.com"
git config user.name "GitHub Actions"
git config user.email "gitea-actions[bot]@noreply.localhost"
git config user.name "Gitea Actions"
- name: Fetch all branches
run: |
git remote set-url origin "https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
git remote set-url origin "https://x-access-token:${PUSH_TOKEN}@gitea.rudolf.keenetic.link/${GITHUB_REPOSITORY}.git"
git fetch --all --prune
- name: Merge every branch onto main
@@ -37,7 +37,6 @@ jobs:
run: |
set -eo pipefail
MAIN=main
# Collect remote branch names (exclude HEAD and main)
branches=$(git for-each-ref --format='%(refname:short)' refs/remotes/origin/ \
| sed 's|^origin/||' \
| grep -v -e '^HEAD$' -e '^main$' -e '^origin$' \
@@ -49,11 +48,8 @@ jobs:
continue
fi
echo "=== Processing branch: $br ==="
# Create/update a local branch from remote
git checkout -B "$br" "origin/$br"
# Merge origin/main into branch (non-interactive)
if git merge --no-edit "origin/$MAIN"; then
# Push updated branch to remote (no force)
if git push origin "$br"; then
echo "Pushed merged $br"
else
+4 -8
View File
@@ -18,16 +18,16 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
token: ${{ secrets.PUSH_TOKEN }}
- name: Configure git
run: |
git config user.email "github-actions[bot]@users.noreply.github.com"
git config user.name "GitHub Actions"
git config user.email "gitea-actions[bot]@noreply.localhost"
git config user.name "Gitea Actions"
- name: Fetch all branches
run: |
git remote set-url origin "https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
git remote set-url origin "https://x-access-token:${PUSH_TOKEN}@gitea.rudolf.keenetic.link/${GITHUB_REPOSITORY}.git"
git fetch --all --prune
- name: Merge every branch onto main
@@ -35,7 +35,6 @@ jobs:
run: |
set -eo pipefail
MAIN=main
# Collect remote branch names (exclude HEAD and main)
branches=$(git for-each-ref --format='%(refname:short)' refs/remotes/origin/ \
| sed 's|^origin/||' \
| grep -v -e '^HEAD$' -e '^main$' -e '^origin$' \
@@ -47,12 +46,9 @@ jobs:
continue
fi
echo "=== Processing branch: $br ==="
# Create/update a local branch from remote
git reset --hard HEAD
git checkout -B "$br" "origin/$br"
# Merge origin/main into branch (non-interactive)
if git merge --no-edit --squash "origin/$MAIN"; then
# Push updated branch to remote (no force)
git reset --soft "origin/$MAIN"
git commit -m "Squash"
if git push origin "$br" --force-with-lease; then
+18 -54
View File
@@ -1,6 +1,3 @@
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
name: Python package
on:
@@ -10,73 +7,40 @@ on:
pull_request:
branches: [ "main" ]
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: [
# "3.11.3",
"3.12.3",
"3.13",
"3.14",
# "3.15",
]
checker: [
"python -c \"import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')\"",
"pip list",
"flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics",
"flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics",
# "ty check backend",
"mypy --no-incremental --config-file mypy.ini backend tests",
]
python-version: ["3.12.3", "3.13", "3.14"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}, ${{ matrix.checker }}
uses: actions/setup-python@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: |
requirements.txt
requirements-dev.txt
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -r requirements-dev.txt
- name: Run checker
run: ${{ matrix.checker }}
tests:
runs-on: ubuntu-latest
needs: build
strategy:
fail-fast: false
matrix:
python-version: [
# "3.11.3",
"3.12.3",
"3.13",
"3.14",
# "3.15",
]
checker: [
"pytest",
]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}, ${{ matrix.checker }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
- name: Lint (flake8)
run: |
# sudo apt-get update
# sudo apt-get upgrade -y
# sudo apt-get install -y --no-install-recommends git docker.io
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
playwright install --with-deps chromium
- name: Run checker
run: "pytest"
flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Type-check (mypy)
run: mypy --no-incremental --config-file mypy.ini backend tests
+111
View File
@@ -0,0 +1,111 @@
# Adding a new stack to brencher
A "stack" is one self-contained Docker Swarm service that brencher manages
end-to-end: detect HEAD changes, build, deploy, and expose service status to
the UI. This file documents how to wire one up so it picks up commits
without an operator clicking "Update" in the UI every time.
## Repository layout
Brencher looks at exactly one commit on one ref per environment. For a new
stack `my_stack`:
1. Push branch `feat/my_stack` (or whatever the team convention is) to a
Git repo listed in `remotes` of the config file.
2. Drop the repo in `gitea` or `github` so it's reachable through the same
remotes that other stacks use.
3. Make sure `my_stack/stack-compose.yml` and a Dockerfile (typically
`my_stack/Dockerfile`) exist at the path the config will point at.
## Config file: `backend/configs/<stack>.py`
The shape is fixed by `enironment.Environment` and the steps it pulls in.
Look at `backend/configs/hermes.py` for the canonical pattern. A new config
must:
- Build a `GitClone` with at least one remote whose `url` points to the repo
above.
- Build a `DockerSwarmCheck(stack_name="<stack>")` to introspect the
deployed service.
- Build **`GitUnmerge(clone, dockerSwarmCheck)`** and wrap it into
`SharedStateHolderInMemory(unmerge=unmerge)`. This is what makes
autodeploy work — `GitUnmerge` reads the `org.brencher.version` label
from the running swarm task and converts it into
`desired_branches`, so each polling cycle asks "is HEAD ≠ what's
deployed?" without any UI input.
- Wire a `CheckoutMerged(clone, desired_branches=state, push=False, …)`.
Push stays `False` for first-party stacks; set `True` only when you want
brencher to push auto-named branches back to origin.
- Build a `DockerComposeBuild` (or `DockerSwarmBuild` if a Swarmfile is
used) pointing at the compose path in the repo.
- Build a `DockerSwarmDeploy(wd=clone, buildDocker=…, stackChecker=…, …)`
with the same `stack_name`. Image tag is `auto-<short-sha>` driven by
`checkoutMerged.progress().version`.
- Optionally append a `SimpleLog(message={"userLinks": {...}})` for the
link to surface in the UI.
- Set `__all__` and export the `Environment` named after the stack id.
- Add it to `Environment.pipeline` in the canonical order:
`clone → state → checkoutMerged → dockerSwarmCheck → build → deploy →
optionals`. Build and deploy are explicit `DockerComposeBuild` and
`DockerSwarmDeploy` siblings; do not collapse them into one step.
If you skip the `GitUnmerge` line, `state.branches` will stay empty and
the stack will only deploy when an operator pushes a `branches` update
through the UI. Don't skip it; it's the difference between a self-driving
stack and a manual one. See `connectivity_tester.py` history for the bug
this caused.
## Verification checklist
- `python3 -c "from backend.configs.my_stack import my_stack"` works in
the brencher container.
- `PROFILES="my_stack" python3 backend/app.py` boots and the UI renders a
row for `my_stack` with non-empty branches (from `GitUnmerge`,
pointing at whatever image is currently deployed).
- Push to the configured branch, wait one full polling cycle
(~`RESET_INTERVAL` = 180s, plus the next pipeline tick), then confirm
the service was redeployed with a new `org.brencher.version` label.
- No UI `environment_update` was issued in between.
## Common foot-guns
- `dockerfile:` path in `stack-compose.yml` is interpreted relative to the
compose file's directory. The compose file and the Dockerfile must
live side-by-side; use `dockerfile: Dockerfile`, not
`dockerfile: <relative-path>`.
- Removing the `name:` key from the top of `stack-compose.yml` is fine
for `docker stack deploy` — the stack name comes from the CLI option
brencher sets.
- `EXTRA_ENDPOINTS` and friends are env vars; if you reference them in
compose, define them in the `envs=lambda: {...}` of the
`DockerComposeBuild` and `DockerSwarmDeploy`. `substitute` uses
`env.get(...)` for deploys but `env[...]` (raises) for builds, so an
unlisted variable kills the build but quietly produces an empty string
in the deploy.
- Compose files must not assume Docker networks that don't exist. Use
either an `external: true` net that already exists in Swarm, drop
the `networks:` block entirely, or `driver: overlay` for a
stack-private net.
## Tracking
When you add a stack, also:
- Open a tracking issue in `admin/brencher` titled
`[ready-to-test] <stack>: …` with `state=` from your
`GitUnmerge`-derived branches and what URL/port the service exposes.
- Add the stack id to the `PROFILES` env var in the brencher stack
compose (`docker_stack` / Swarm declarative) so it gets picked up on
next rollout. The admin overlay is where this lives.
## Build delegation pattern (Actions)
Brencher does not run docker build or docker push itself. It dispatches a Gitea
Actions workflow via `POST /api/v1/repos/{owner}/{repo}/actions/workflows/{file}/dispatches`,
waits for the run to complete, then runs `docker stack deploy`.
Use `steps.actions.GiteaActionsTrigger` and `steps.actions.GiteaActionsAwait`
in the stack config, set `buildDocker=None` on `DockerSwarmDeploy`, and let Swarm
resolve and pull the image on target nodes during deploy. Credentials must be
available to the deploy path and the compose image must point to a registry the
runner can push to (e.g. Gitea Container Registry `gitea.rudolf.keenetic.link/<owner>/<repo>`).
+1 -1
View File
@@ -33,7 +33,7 @@ docker service update --secret-rm brencher-secrets brencher_brencher-backend
docker service update --secret-rm brencher-secrets brencher2_brencher-backend
docker secret rm brencher-secrets
printf "GIT_USERNAME=git\nGIT_PASSWORD=TODO\n" | \
printf "GIT_USERNAME=git\nGIT_PASSWORD=TODO\nGITEA_TOKEN=TODO\n" | \
docker secret create brencher-secrets -
docker service update --secret-add brencher-secrets brencher_brencher-backend
+3
View File
@@ -17,3 +17,6 @@
state: str # 'Active' or 'Pause'
pipeline: Dict[str, Any]
+2 -2
View File
@@ -48,7 +48,7 @@ class App:
"name": r.name,
"status": [str(result), stack],
"error": True,
"is_running": True,
"is_running": False,
})
else:
pipeline_state.append({
@@ -62,7 +62,7 @@ class App:
"name": r.name,
"status": [str(e), stack],
"error": True,
"is_running": True,
"is_running": False,
})
env_dtos[env.id] = {'id': env.id }
env_dtos[env.id]['pipeline'] = pipeline_state
+10 -6
View File
@@ -5,7 +5,11 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/brencher.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/brencher"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/brencher"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher",
@@ -45,10 +49,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="docker-compose.yml",
)
checkPing = UrlCheck(
url="https://brencher.rudolf.keenetic.link/state",
expected=checkPingF,
)
#checkPing = UrlCheck(
# url="https://brencher.rudolf.keenetic.link/state",
# expected=checkPingF,
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://brencher.rudolf.keenetic.link/",
@@ -68,7 +72,7 @@ brencher = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
]
)
+10 -6
View File
@@ -21,7 +21,11 @@ def checkPingF(obj: Any) -> None:
raise Exception(f"brencher2 check failed for: {v}")
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/brencher.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/brencher"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/brencher"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher2",
@@ -72,10 +76,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="docker-compose.yml",
)
checkPing = UrlCheck(
url="https://brencher.rudolf.keenetic.link/state",
expected=checkPingF,
)
#checkPing = UrlCheck(
# url="https://brencher.rudolf.keenetic.link/state",
# expected=checkPingF,
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://brencher.rudolf.keenetic.link/",
@@ -98,7 +102,7 @@ brencher2 = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
],
)
+93
View File
@@ -0,0 +1,93 @@
import os
from enironment import Environment
from steps.actions import GiteaActionsAwait, GiteaActionsTrigger
from steps.checks import SimpleLog
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import CheckoutMerged, GitClone, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(
remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/swarm_connectivity_tester"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/swarm_connectivity_tester"},
{"name": "gitea_org1", "url": "https://gitea.rudolf.keenetic.link/org1/swarm_connectivity_tester"},
],
credEnvPrefix="GITEA",
)
dockerSwarmCheck = DockerSwarmCheck(stack_name="connectivity_tester")
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(
clone,
desired_branches=state,
push="gitea_org1",
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot",
)
GITEA_API = os.environ.get("GITEA_API", "https://gitea.rudolf.keenetic.link/api/v1")
GITEA_TOKEN = os.environ.get("GITEA_TOKEN", "")
WORKFLOW_OWNER = "org1"
WORKFLOW_REPO = "swarm_connectivity_tester"
WORKFLOW_FILE = "release.yml"
def _inputs() -> dict:
return {"tag": "auto-" + checkoutMerged.progress().version}
trigger_build = GiteaActionsTrigger(
api_base=GITEA_API,
owner=WORKFLOW_OWNER,
repo=WORKFLOW_REPO,
workflow_file=WORKFLOW_FILE,
ref=lambda: checkoutMerged.progress().remote_branch_name or "master",
token=GITEA_TOKEN,
inputs=_inputs,
wd=checkoutMerged,
)
await_build = GiteaActionsAwait(
api_base=GITEA_API,
owner=WORKFLOW_OWNER,
repo=WORKFLOW_REPO,
token=GITEA_TOKEN,
run=trigger_build,
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=None,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
"registry_owner": "org1",
},
stack_name="connectivity_tester",
docker_compose_path="stack-compose.yml",
)
logUrls = SimpleLog(message={
"userLinks": {
"Connectivity tester": "http://connectivity-tester.keenetic.link:8765/",
},
})
__all__ = ["connectivity_tester"]
connectivity_tester = Environment(
id="connectivity_tester",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
trigger_build,
await_build,
deployDocker,
logUrls,
],
)
+80
View File
@@ -0,0 +1,80 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="gitea",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=None,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="gitea",
docker_compose_path="poc/gitea/stack-compose.yml",
)
# checkPing1 = UrlCheck(
# url="http://100.70.193.97:8087/api/status",
# expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
# )
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://gitea.rudolf.keenetic.link",
}
})
__all__ = ["gitea"]
gitea = Environment(
id="gitea",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
# checkPing1,
logUrls
]
)
+6 -3
View File
@@ -4,7 +4,10 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="immoscout")
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/mcps"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/mcps"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="gmail_mcp",
@@ -23,7 +26,7 @@ checkoutMerged = CheckoutMerged(clone,
buildDocker = DockerComposeBuild(clone,
docker_repo_username="",
docker_repo_password="",
docker_compose_path="gmail-mcp-server/docker-compose.yml",
docker_compose_path="docker-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
envs=lambda: {
@@ -39,7 +42,7 @@ deployDocker = DockerSwarmDeploy(
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="gmail_mcp",
docker_compose_path="gmail-mcp-server/docker-compose.yml",
docker_compose_path="docker-compose.yml",
)
checkPing1 = UrlCheck(
+63
View File
@@ -0,0 +1,63 @@
from enironment import Environment
from steps.checks import SimpleLog
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(remotes=[
# {"name": "github", "url": "https://github.com/rudolf1/uber_backup.git", "prefix": "ansible"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="hermes",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=None,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="hermes",
docker_compose_path="poc/hermes/stack-compose.yml",
)
# checkPing1 = UrlCheck(
# url="http://100.70.193.97:8087/api/status",
# expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
# )
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://hermes.rudolf.keenetic.link",
}
})
__all__ = ["hermes"]
hermes = Environment(
id="hermes",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
# checkPing1,
logUrls
]
)
+94
View File
@@ -0,0 +1,94 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="hermes_infra",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
buildDocker = DockerComposeBuild(clone,
docker_repo_username="",
docker_repo_password="",
docker_compose_path="hermes_squid/stack-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
build_cache = True,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
# "user_group": "1000:137"
},
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=buildDocker,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="hermes_infra",
docker_compose_path="hermes_squid/stack-compose.yml",
)
# checkPing1 = UrlCheck(
# url="http://100.70.193.97:8088/api/status",
# expected=lambda obj: obj['gateway_running'] == 'true' and obj['auth_required'] == 'true',
# )
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://hermes.rudolf.keenetic.link",
}
})
__all__ = ["hermes_infra"]
hermes_infra = Environment(
id="hermes_infra",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
deployDocker,
# checkPing1,
logUrls
]
)
+27 -6
View File
@@ -4,7 +4,28 @@ from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")
# Dual-remote clone for the ansible mirror (the deploy target, not brencher itself).
# - gitea_hermes is where the feature branch lives (hermes/ansible:*)
# - gitea_admin is the live-deploy source (where brencher-bot reads compose files from)
#
# Why not github? github.com is unreachable from the brencher container -- the
# proxy returns HTTP 403 on CONNECT tunnel, so the URL would always fail to
# fetch. See backend/configs/docker_socket_proxy.py for the same migration.
#
# Why drop branchNamePrefix? GitClone only applies its prefix refspec to a
# remote named "origin" (backend/steps/git.py ~L104); neither remote here is
# named "origin", so the prefix would be a silent no-op. Both remotes fetch
# their full refset instead.
#
# Why two remotes? The deploy step (DockerSwarmDeploy) checks out a specific
# branch and uses the resulting checkout as wd for the compose path. With
# gitea_hermes as primary, feature branches opened in the fork are picked up;
# gitea_admin provides the live-deploy source as a fallback / for cross-fork
# PRs. This matches the form in backend/configs/brencher.py after PR #3.
clone = GitClone(remotes=[
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/ansible"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/ansible"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="immich",
@@ -31,10 +52,10 @@ deployDocker = DockerSwarmDeploy(
docker_compose_path="poc/immich/stack-compose.yml",
)
checkPing = UrlCheck(
url="https://immich.rudolf.keenetic.link/api/server/ping",
expected={"res": "pong"},
)
#checkPing = UrlCheck(
# url="https://immich.rudolf.keenetic.link/api/server/ping",
# expected={"res": "pong"},
#)
logUrls = SimpleLog(message={
"userLinks": {
"App": "https://immich.rudolf.keenetic.link",
@@ -52,7 +73,7 @@ immich = Environment(
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing,
# checkPing,
logUrls
]
)
+5 -1
View File
@@ -3,7 +3,11 @@ from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/reverseproxy.git")
clone = GitClone(remotes=[
{"name": "github", "url": "https://github.com/rudolf1/reverseproxy.git"},
{"name": "gitea_admin", "url": "https://gitea.rudolf.keenetic.link/admin/reverseproxy"},
{"name": "gitea_hermes", "url": "https://gitea.rudolf.keenetic.link/hermes/reverseproxy"},
])
dockerSwarmCheck = DockerSwarmCheck(
stack_name="torrserv_proxy",
+215
View File
@@ -0,0 +1,215 @@
import json
import logging
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from typing import Any, Callable, Dict, Optional, Union
from enironment import AbstractStep
from steps.git import CheckoutMerged
logger = logging.getLogger(__name__)
@dataclass
class GiteaActionsRun:
run_id: int
status: str
conclusion: Optional[str]
html_url: str
InputsSource = Union[Dict[str, str], Callable[[], Dict[str, str]], None]
RefSource = Union[str, Callable[[], Optional[str]], None]
class GiteaActionsTrigger(AbstractStep[GiteaActionsRun]):
"""Trigger a Gitea Actions workflow via the dispatches API."""
def __init__(self,
api_base: str,
owner: str,
repo: str,
workflow_file: str,
ref: RefSource,
token: str,
inputs: InputsSource = None,
wd: Optional[CheckoutMerged] = None,
poll_timeout_sec: int = 1800,
poll_interval_sec: int = 10,
**kwargs: Any) -> None:
super().__init__(**kwargs)
self.api_base = api_base.rstrip("/")
self.owner = owner
self.repo = repo
self.workflow_file = workflow_file
self.ref = ref
self.token = token
self.inputs = inputs
self.wd = wd
self.poll_timeout_sec = poll_timeout_sec
self.poll_interval_sec = poll_interval_sec
self._last_request_key: tuple[Optional[str], tuple[tuple[str, str], ...]] | None = None
self._last_run: GiteaActionsRun | None = None
def _http(self, method: str, url: str, body: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
data = None
headers = {"Authorization": f"token {self.token}", "Accept": "application/json"}
if body is not None:
data = json.dumps(body).encode()
headers["Content-Type"] = "application/json"
req = urllib.request.Request(url, data=data, headers=headers, method=method)
with urllib.request.urlopen(req, timeout=30) as r:
body_bytes = r.read()
if not body_bytes or r.status == 204:
return {}
return json.loads(body_bytes.decode())
def _list_recent_runs(self) -> Optional[GiteaActionsRun]:
url = f"{self.api_base}/repos/{self.owner}/{self.repo}/actions/runs?limit=1"
try:
data = self._http("GET", url)
except urllib.error.HTTPError as e:
logger.error("list runs failed: %s", e)
return None
items = data.get("workflow_runs") or []
if not items:
return None
r = items[0]
return GiteaActionsRun(
run_id=r["id"],
status=r.get("status", ""),
conclusion=r.get("conclusion"),
html_url=r.get("html_url", ""),
)
def progress(self) -> GiteaActionsRun:
if self.env.dry:
logger.info("DRY RUN: Skipping Gitea Actions workflow dispatch.")
return GiteaActionsRun(
run_id=0,
status="dry-run",
conclusion=None,
html_url="",
)
resolved_inputs: Dict[str, str] = {}
if callable(self.inputs):
resolved_inputs = self.inputs() or {}
elif isinstance(self.inputs, dict):
resolved_inputs = self.inputs
ref = self.ref() if callable(self.ref) else self.ref
request_key = (ref, tuple(sorted(resolved_inputs.items())))
latest = self._list_recent_runs()
if latest is not None and latest.status != "completed":
logger.info("Gitea Actions run %s is already active; reusing it.", latest.run_id)
self._last_request_key = request_key
self._last_run = latest
return latest
if (
self._last_request_key == request_key
and self._last_run is not None
and latest is not None
and latest.run_id == self._last_run.run_id
and latest.conclusion == "success"
):
logger.info("Gitea Actions run %s already succeeded; reusing it.", latest.run_id)
return latest
body: Dict[str, Any] = {"ref": ref}
if resolved_inputs:
body["inputs"] = resolved_inputs
url = (
f"{self.api_base}/repos/{self.owner}/{self.repo}"
f"/actions/workflows/{self.workflow_file}/dispatches"
)
try:
self._http("POST", url, body=body)
except urllib.error.HTTPError as e:
if e.code != 500:
raise
raise RuntimeError(
"POST /dispatches returned 500 (gitea-side bug on this instance). "
"Workaround: trigger the workflow manually by pushing a tag matching "
f"the workflow's `on.push.tags` filter, e.g. "
f"`git push <repo> HEAD:refs/tags/auto-<sha>`. "
f"Requested workflow: {self.owner}/{self.repo}/{self.workflow_file}@{ref}"
) from e
deadline = time.time() + self.poll_timeout_sec
while time.time() < deadline:
current = self._list_recent_runs()
if current is None or (latest is not None and current.run_id == latest.run_id):
time.sleep(self.poll_interval_sec)
continue
self._last_request_key = request_key
self._last_run = current
return current
raise TimeoutError(
f"Gitea Actions did not pick up a new run within {self.poll_timeout_sec}s"
)
class GiteaActionsAwait(AbstractStep[GiteaActionsRun]):
"""Wait for a previously-triggered Gitea Actions run to complete."""
def __init__(self,
api_base: str,
owner: str,
repo: str,
token: str,
run: Union[AbstractStep[GiteaActionsRun], Callable[[], GiteaActionsRun]],
poll_timeout_sec: int = 1800,
poll_interval_sec: int = 10,
**kwargs: Any) -> None:
super().__init__(**kwargs)
self.api_base = api_base.rstrip("/")
self.owner = owner
self.repo = repo
self.token = token
self._run = run
self.poll_timeout_sec = poll_timeout_sec
self.poll_interval_sec = poll_interval_sec
def _http(self, method: str, url: str) -> Dict[str, Any]:
headers = {"Authorization": f"token {self.token}", "Accept": "application/json"}
req = urllib.request.Request(url, headers=headers, method=method)
with urllib.request.urlopen(req, timeout=30) as r:
return json.loads(r.read().decode())
def progress(self) -> GiteaActionsRun:
if self.env.dry:
logger.info("DRY RUN: Skipping wait for Gitea Actions workflow.")
return GiteaActionsRun(
run_id=0,
status="dry-run",
conclusion=None,
html_url="",
)
run = self._run.progress() if isinstance(self._run, AbstractStep) else self._run()
data = self._http(
"GET",
f"{self.api_base}/repos/{self.owner}/{self.repo}/actions/runs/{run.run_id}",
)
status = data.get("status", "")
conclusion = data.get("conclusion")
if status != "completed":
raise RuntimeError(
f"Gitea Actions run {run.run_id} is not completed: {status}: "
f"{run.html_url}"
)
if conclusion != "success":
raise RuntimeError(
f"Gitea Actions run {run.run_id} concluded with {conclusion}: "
f"{run.html_url}"
)
return GiteaActionsRun(
run_id=run.run_id,
status=status,
conclusion=conclusion,
html_url=run.html_url,
)
+71 -5
View File
@@ -1,3 +1,4 @@
import hashlib
import logging
import os
import re
@@ -197,12 +198,39 @@ class DockerSwarmDeploy(AbstractStep[str]):
content = f.read()
content = re.sub(r'\$\{([^}]+)\}', lambda m: env.get(m.group(1), ""), content)
compose = yaml.safe_load(content)
version = env["version"]
if "configs" in compose:
sanitized_version = re.sub(r'[^a-zA-Z0-9_.-]', '_', version)
full_prefix = f"{self.stack_name}_"
def _config_full_name(base_name: str) -> str:
full = f"{full_prefix}{base_name}_{sanitized_version}"
if len(full) <= 64:
return f"{base_name}_{sanitized_version}"
return hashlib.sha256(full.encode("utf-8")).hexdigest()[:32]
config_names = {
name: _config_full_name(name)
for name in compose["configs"]
}
compose["configs"] = {
config_names.get(name, name): config
for name, config in compose["configs"].items()
}
for service in compose.get("services", {}).values():
for config in service.get("configs", []):
if isinstance(config, str):
config_index = service["configs"].index(config)
service["configs"][config_index] = config_names.get(config, config)
elif isinstance(config, dict) and "source" in config:
config["source"] = config_names.get(config["source"], config["source"])
if "services" in compose:
for svc in compose["services"].values():
if "build" in svc:
del svc["build"]
svc["labels"] = {"org.brencher.version": env["version"]}
del env["version"]
env.pop("registry_owner", None)
merge_dicts(compose, env)
# logger.info(f"Final compose: {compose}")
content = yaml.safe_dump(compose)
@@ -268,6 +296,8 @@ class DockerSwarmDeploy(AbstractStep[str]):
# Deploy stack to Docker Swarm
cmd = [
"docker", "stack", "deploy",
"--with-registry-auth",
"--detach=false",
"--prune",
"-c", tmp_compose_path,
self.stack_name
@@ -279,12 +309,27 @@ class DockerSwarmDeploy(AbstractStep[str]):
swarmEnv = {k: v for k, v in dotenv_values(os.path.join(os.path.dirname(tmp_compose_path), ".env")).items()
if v is not None}
# merge_dicts(swarmEnv, env)
result = subprocess.run(cmd, capture_output=True, text=True)
# , cwd=os.path.dirname(tmp_compose_path), env=swarmEnv)
result = subprocess.run(cmd, capture_output=True, text=True, cwd=os.path.dirname(tmp_compose_path))
#, env=swarmEnv)
os.remove(tmp_compose_path)
if result.returncode != 0:
logger.error(f"Stack deploy failed: {result.stderr}")
raise RuntimeError(f"Stack deploy failed: {result.stderr}")
logger.info(f"docker stack deploy stdout: {result.stdout or '<empty>'}")
logger.info(f"docker stack deploy stderr: {result.stderr or '<empty>'}")
stderr_lower = result.stderr.lower()
docker_error_markers = (
"failed to ",
"error response from daemon",
"rpc error",
"denied",
"no such image",
)
if result.returncode != 0 or any(marker in stderr_lower for marker in docker_error_markers):
deploy_output = (
f"exit code: {result.returncode}\n"
f"stdout:\n{result.stdout or '<empty>'}\n"
f"stderr:\n{result.stderr or '<empty>'}"
)
logger.error(f"Stack deploy failed:\n{deploy_output}")
raise RuntimeError(f"Stack deploy failed:\n{deploy_output}")
logger.info(f"Stack deployed successfully: {result.stdout}")
# Check health of all deployed services
@@ -308,4 +353,25 @@ class DockerSwarmDeploy(AbstractStep[str]):
if not_running:
raise BaseException(f"Services not started yet: {', '.join(not_running)}")
version_mismatches = []
for svc_name, svc in expected_services.items():
full_svc_name = f"{self.stack_name}_{svc_name}"
services_list = client.services.list(filters={"name": full_svc_name})
if not services_list:
continue
service_labels = services_list[0].attrs.get("Spec", {}).get("TaskTemplate", {}).get("ContainerSpec", {}).get("Labels", {})
expected_version = svc.get("labels", {}).get("org.brencher.version")
actual_version = service_labels.get("org.brencher.version")
if expected_version != actual_version:
version_mismatches.append(
f"{full_svc_name} expected version {expected_version}, got {actual_version}"
)
update_status = services_list[0].attrs.get("UpdateStatus", {})
if update_status.get("State") in {"paused", "rollback_paused", "rollback_started"}:
version_mismatches.append(
f"{full_svc_name} Swarm update {update_status.get('State')}: {update_status.get('Message', '')}"
)
if version_mismatches:
raise RuntimeError(f"Deployment did not apply requested versions: {'; '.join(version_mismatches)}")
return f"Stack deployed successfully: {result.stdout}"
+49
View File
@@ -0,0 +1,49 @@
import json
import logging
import os
import re
import subprocess
from typing import Any, Callable, Dict
import yaml
from enironment import AbstractStep
from steps.git import GitClone
logger = logging.getLogger(__name__)
class DockerComposePull(AbstractStep[Dict[str, str]]):
"""Pull images referenced by a docker-compose file from a remote registry."""
def __init__(self,
wd: GitClone,
docker_compose_path: str,
envs: Callable[[], Dict[str, Any]],
**kwargs: Any) -> None:
super().__init__(**kwargs)
self.wd = wd
self.envs = envs
self.docker_compose_path = docker_compose_path
def progress(self) -> Dict[str, str]:
env = self.envs()
compose_path = os.path.join(self.wd.progress(), self.docker_compose_path)
with open(compose_path, "r") as f:
content = f.read()
content = re.sub(r"\$\{([^}]+)\}", lambda m: env.get(m.group(1), ""), content)
compose = yaml.safe_load(content)
for svc in compose.get("services", {}).values():
if "build" in svc:
del svc["build"]
tmp_path = compose_path + ".pull.yml"
with open(tmp_path, "w") as f:
yaml.safe_dump(compose, f)
try:
subprocess.run(
["docker", "compose", "-f", tmp_path, "pull"],
check=True,
cwd=self.wd.progress(),
)
finally:
os.unlink(tmp_path)
return {name: svc.get("image", "") for name, svc in compose.get("services", {}).items()}
+128 -50
View File
@@ -16,11 +16,57 @@ logger = logging.getLogger(__name__)
class GitClone(AbstractStep[str]):
def __init__(self, url: str, repo_path: str | None = None,
branchNamePrefix: str = "", credEnvPrefix: str = "GIT",
def __init__(self,
url: str | List[str] | None = None,
remotes: List[Dict[str, str]] | None = None,
repo_path: str | None = None,
branchNamePrefix: str = "",
credEnvPrefix: str = "GIT",
**kwargs: Any):
super().__init__(**kwargs)
self.url = url
# `remotes` is the explicit form: a list of {"name": "...", "url": "..."}.
# `url` is kept as a shorthand: a single string becomes one remote named
# "origin"; a list of strings becomes remotes named "origin", "repo1",
# "repo2", ... (matches the behavior introduced by the "multiple remotes"
# change so existing positional-list callers keep working).
if remotes is not None and url is not None:
raise ValueError("Pass either 'url' or 'remotes', not both")
normalized: List[Dict[str, str]] = []
if remotes is not None:
seen: Set[str] = set()
for entry in remotes:
if not isinstance(entry, dict):
raise TypeError(
f"Each entry in 'remotes' must be a dict with 'name' and 'url', got {type(entry).__name__}"
)
name = entry.get("name")
url_value = entry.get("url")
if not name or not isinstance(name, str):
raise ValueError("Each remote requires a non-empty 'name' (string)")
if not url_value or not isinstance(url_value, str):
raise ValueError(f"Remote {name!r} requires a non-empty 'url' (string)")
if name in seen:
raise ValueError(f"Duplicate remote name: {name!r}")
seen.add(name)
normalized.append({"name": name, "url": url_value})
elif url is not None:
if isinstance(url, str):
raw = [url]
else:
raw = list(url)
for index, repository_url in enumerate(raw):
name = "origin" if index == 0 else f"repo{index}"
normalized.append({"name": name, "url": repository_url})
else:
raise ValueError("GitClone requires either 'url' or 'remotes'")
if len(normalized) == 0:
raise ValueError("At least one repository URL is required")
self.remotes = normalized
# Derived view used by repo_path keying and any introspection.
self.urls = [r["url"] for r in normalized]
self.repo_path = repo_path
self.branchNamePrefix = branchNamePrefix
self.credEnvPrefix = credEnvPrefix
@@ -36,33 +82,42 @@ class GitClone(AbstractStep[str]):
return url
def progress(self) -> str:
repo_url = self.url
self.repo_path = self.repo_path or os.path.join(tempfile.gettempdir(),
f"{self.env.id}_{hashlib.sha1(repo_url.encode()).hexdigest()[:5]}")
logger.info(f"Cloning repository {repo_url} to {self.repo_path}")
f"{self.env.id}_{hashlib.sha1('|'.join(self.urls).encode()).hexdigest()[:5]}")
logger.info(f"Cloning repositories {self.remotes} to {self.repo_path}")
os.makedirs(self.repo_path, exist_ok=True)
try:
if os.path.exists(os.path.join(self.repo_path, ".git")):
logger.info(f"Repository already cloned at {self.repo_path}, fetching updates.")
repo = git.Repo(self.repo_path)
if 'origin' not in repo.remotes or repo.remotes.origin.url != self._get_auth_git_url(repo_url):
repo.create_remote('origin', self._get_auth_git_url(repo_url))
result = repo.remotes.origin.fetch(prune=True)
if not result or any(fetch_info.flags & fetch_info.ERROR for fetch_info in result):
raise BaseException(f"Failed to fetch updates for {repo_url}")
else:
repo = git.Repo.init(self.repo_path)
repo.remotes.append(repo.create_remote(
'origin',
self._get_auth_git_url(repo_url)
))
if self.branchNamePrefix != "":
repo.config_writer().set_value('remote "origin"', "fetch",
f"+refs/heads/{self.branchNamePrefix}/*:refs/remotes/origin/{self.branchNamePrefix}/*").release()
repo.remotes.origin.fetch(prune=True)
if not os.path.exists(os.path.join(self.repo_path, ".git")):
raise BaseException(f"Failed to clone repository {repo_url} to {self.repo_path}")
for entry in self.remotes:
remote_name = entry["name"]
repository_url = entry["url"]
auth_url = self._get_auth_git_url(repository_url)
if remote_name in repo.remotes:
remote = repo.remotes[remote_name]
if remote.url != auth_url:
remote.set_url(auth_url)
else:
remote = repo.create_remote(remote_name, auth_url)
with repo.config_writer() as config:
config.set_value(
f'remote "{remote_name}"',
'fetch',
f'+refs/heads/*:refs/remotes/{remote_name}/*',
)
if remote_name == "origin" and self.branchNamePrefix != "":
with repo.config_writer() as config:
config.set_value('remote "origin"', "fetch",
f"+refs/heads/{self.branchNamePrefix}/*:refs/remotes/origin/{self.branchNamePrefix}/*")
result = remote.fetch(prune=True)
if not result or any(fetch_info.flags & fetch_info.ERROR for fetch_info in result):
raise BaseException(f"Failed to fetch updates for {remote_name}")
if not os.path.exists(os.path.join(self.repo_path, ".git")):
raise BaseException(f"Failed to clone repositories {self.remotes} to {self.repo_path}")
except BaseException as e:
logger.error(f"Error during git clone/fetch, removing directory {self.repo_path}: {str(e)}")
shutil.rmtree(self.repo_path)
@@ -73,21 +128,24 @@ class GitClone(AbstractStep[str]):
repo = git.Repo(self.repo_path)
result: Dict[str, List[Any]] = {}
for ref in repo.refs:
if ref.name.startswith('origin/') and not ref.name.startswith('origin/HEAD'):
branch_name = ref.name[len('origin/'):]
if not branch_name.startswith('auto/'): # Skip auto branches
result[branch_name] = []
cmt = [ref.commit]
for _ in range(10):
for commit in cmt:
result[branch_name].append({
'hexsha': commit.hexsha,
'author': commit.author.name,
'date': commit.committed_datetime.isoformat(),
'message': commit.message.strip()
})
if not ref.is_remote() or ref.name.endswith('/HEAD'):
continue
remote_name, branch_name = ref.name.split('/', 1)
if branch_name.startswith('auto/'): # Skip auto branches
continue
branch_key = branch_name if remote_name == 'origin' else ref.name
result[branch_key] = []
cmt = [ref.commit]
for _ in range(10):
for commit in cmt:
result[branch_key].append({
'hexsha': commit.hexsha,
'author': commit.author.name,
'date': commit.committed_datetime.isoformat(),
'message': commit.message.strip()
})
cmt = [p for c in cmt for p in c.parents]
cmt = [p for c in cmt for p in c.parents]
return result
@@ -112,6 +170,18 @@ def ensure_clean(repo: git.Repo) -> None:
raise BaseException(f"Changes in repo: U{repo.untracked_files}")
def _resolve_branch_ref(repo: git.Repo, branch_key: str, default_remote: str) -> str:
# Branch keys can be "branch", "origin/branch", "gitea_admin/branch", etc.
# If the key has a remote prefix that matches a known remote, use it
# verbatim; otherwise fall back to the primary configured remote so
# configs that don't name a remote `origin` still work.
if "/" in branch_key:
remote_name, _ = branch_key.split("/", 1)
if remote_name in repo.remotes:
return branch_key
return f"{default_remote}/{branch_key}"
class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
wd: GitClone
@@ -119,7 +189,7 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
desired_branches: AbstractStep[SharedState],
git_user_email: str,
git_user_name: str,
push: bool = True,
push: bool | str = True,
**kwargs: Any):
super().__init__(**kwargs)
self.wd = wd
@@ -146,11 +216,19 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
def find_desired_commits(self, repo: git.Repo, branches: List[Tuple[str, str]]) -> Dict[Commit, str]:
# Extract commit ids for the selected branches
commit_ids: Dict[Commit, str] = {}
# Default remote: prefer the first remote GitClone was configured with
# (so non-"origin" configs like github/gitea_admin work without the
# caller having to qualify every branch key).
default_remote = "origin"
if isinstance(getattr(self.wd, "remotes", None), list) and self.wd.remotes:
primary = self.wd.remotes[0]["name"]
if primary in repo.remotes:
default_remote = primary
for branch_pair in branches:
try:
branch_name, desired_commit = branch_pair
if desired_commit == 'HEAD':
commit = repo.commit(f'origin/{branch_name}')
commit = repo.commit(_resolve_branch_ref(repo, branch_name, default_remote))
else:
commit = repo.commit(desired_commit)
commit_ids[commit] = branch_name
@@ -229,16 +307,18 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
version = '-'.join([x.hexsha[0:8] for x in sorted_commits])
for ref in repo.refs:
if ref.is_remote() and ref.commit == repo.head.commit and ref.name != 'origin/HEAD':
if ref.is_remote() and ref.commit == repo.head.commit and not ref.name.endswith('/HEAD'):
logger.info(f"Merge commit {ref.commit} corresponds to branch {ref}")
remote_branch_name = ref.name[len('origin/'):]
remote_name, branch_name = ref.name.split('/', 1)
remote_branch_name = branch_name if remote_name == 'origin' else ref.name
break
if self.push and remote_branch_name is None:
if self.push:
auto_branch_hash = hashlib.sha1(''.join([x.hexsha for x in sorted_commits]).encode()).hexdigest()
auto_branch_name = f"auto-{version}"
push_remote = self.push if isinstance(self.push, str) else "origin"
logger.info(f"Pushing {repo.head.commit.hexsha} -> {auto_branch_name}")
repo.git.push('-f', 'origin', f"HEAD:refs/heads/{auto_branch_name}")
repo.git.push('-f', push_remote, f"HEAD:refs/heads/{auto_branch_name}")
remote_branch_name = auto_branch_name
return CheckoutAndMergeResult(
@@ -292,10 +372,9 @@ class GitUnmerge(AbstractStep[GitUnmergeResult]):
commits: List[Tuple[str, str]] = []
for v in version_parts:
commit = repo.commit(v)
# branches = [head.name for head in repo.heads if head.commit.hexsha == commit.hexsha]
branches = [ref.name for ref in repo.remotes.origin.refs if ref.commit.hexsha == commit.hexsha]
branches = [b[len('origin/'):] for b in branches if
b.startswith('origin/') and not b.startswith('origin/HEAD')]
branches = [ref.name for ref in repo.refs
if ref.is_remote() and not ref.name.endswith('/HEAD') and ref.commit.hexsha == commit.hexsha]
branches = [b[len('origin/'):] if b.startswith('origin/') else b for b in branches]
if len(branches) == 0 or branches[0].startswith('auto/'):
if childs is None:
@@ -304,10 +383,9 @@ class GitUnmerge(AbstractStep[GitUnmergeResult]):
while len(commitsSet) > 0:
current = commitsSet.pop()
for child in childs.get(current, []):
branches = [ref.name for ref in repo.remotes.origin.refs if
ref.commit.hexsha == child.hexsha]
branches = [b[len('origin/'):] for b in branches if
b.startswith('origin/') and not b.startswith('origin/HEAD')]
branches = [ref.name for ref in repo.refs
if ref.is_remote() and not ref.name.endswith('/HEAD') and ref.commit.hexsha == child.hexsha]
branches = [b[len('origin/'):] if b.startswith('origin/') else b for b in branches]
if len(branches) > 0:
commitsSet = set()
+5 -2
View File
@@ -42,8 +42,11 @@ class SharedStateHolderInMemory(AbstractStep[SharedState], SharedStateHolder):
def progress(self) -> SharedState:
if len(self.state.branches) == 0:
if self.unmerge:
unmerge = self.unmerge.progress()
self.state = replace(self.state, branches=unmerge.branches, token=uuid.uuid4().hex)
try:
unmerge = self.unmerge.progress()
self.state = replace(self.state, branches=unmerge.branches, token=uuid.uuid4().hex)
except BaseException as e:
logger.error("Unmerge failed: %s", e)
return self.state
+32 -8
View File
@@ -94,6 +94,15 @@ class WebApp:
merged = _merge_dicts(merged, connector.branches)
return merged
def get_selection_summary(self) -> Dict[str, Dict[str, Any]]:
return {
env_id: {
"branches": env.get("branches", []),
"dry": env.get("dry"),
}
for env_id, env in self.get_global_envs_to_emit().items()
}
# --- Async scheduling from sync threads ---
def _schedule_async(self, coro: Any) -> None:
@@ -107,12 +116,12 @@ class WebApp:
disconnected = set()
message = custom_json_dumps({event: data})
for websocket in self.ws_connections.keys():
if self.ws_connections[websocket].get(event) == message:
for websocket, state in self.ws_connections.items():
if state.get(event) == message:
continue
try:
await websocket.send_text(message)
self.ws_connections[websocket][event] = message
state[event] = message
except Exception as e:
logger.error(f"Error sending to websocket: {e}")
disconnected.add(websocket)
@@ -129,6 +138,7 @@ class WebApp:
async def broadcast_environments(self, data: Any) -> None:
await self.broadcast("environments", data)
await self.broadcast("version", self.get_version())
async def broadcast_error(self, data: Any) -> None:
await self.broadcast("error", data)
@@ -140,11 +150,16 @@ class WebApp:
# --- Routes ---
def get_version(self) -> str:
return os.getenv('VERSION', 'unknown') or 'unknown'
async def serve_index(self) -> FileResponse:
return FileResponse(os.path.join(FRONTEND_DIR, 'index.html'))
async def serve_state(self) -> Response:
return Response(content=custom_json_dumps(self.get_global_envs_to_emit()), media_type="application/json")
state = self.get_global_envs_to_emit()
state['version'] = self.get_version()
return Response(content=custom_json_dumps(state), media_type="application/json")
async def serve_branches_route(self) -> Response:
return Response(content=custom_json_dumps(self.get_global_branches_to_emit()), media_type="application/json")
@@ -157,16 +172,19 @@ class WebApp:
async def websocket_endpoint(self, websocket: WebSocket) -> None:
await websocket.accept()
self.ws_connections[websocket] = {}
state = self.ws_connections.setdefault(websocket, {})
logger.info("UI connected: selected=%s", self.get_selection_summary())
try:
# Send initial state on connect and record it so duplicate broadcasts are suppressed
branches_payload = custom_json_dumps({"branches": self.get_global_branches_to_emit()})
envs_payload = custom_json_dumps({"environments": self.get_global_envs_to_emit()})
version_payload = custom_json_dumps({"version": self.get_version()})
await websocket.send_text(version_payload)
await websocket.send_text(branches_payload)
self.ws_connections[websocket]["branches"] = branches_payload
state["branches"] = branches_payload
await websocket.send_text(envs_payload)
self.ws_connections[websocket]["environments"] = envs_payload
state["environments"] = envs_payload
while True:
data = await websocket.receive_text()
@@ -174,7 +192,12 @@ class WebApp:
if "update" in message:
update_data = message.get("update") or {}
logger.info(f"Received environment update: {update_data}")
logger.info(
"UI selection update: env=%s branches=%s dry=%s",
update_data.get('id', ''),
update_data.get('branches'),
update_data.get('dry'),
)
if self.secondaryManager:
await self.secondaryManager.send({"update": update_data})
id = update_data.get('id', '')
@@ -214,6 +237,7 @@ class WebApp:
if isinstance(p, CachingStep):
p.reset()
logger.info(f"Updated environment {env.id} branches to {update_data.get('branches')}, dry={update_data.get('dry')}")
logger.info("UI selection resolved: selected=%s", self.get_selection_summary())
self.core.environment_update_event.set()
await self.broadcast_environments(self.get_global_envs_to_emit())
+2 -1
View File
@@ -16,7 +16,8 @@ services:
- PYTHONUNBUFFERED=1
- VIRTUAL_HOST=brencher.rudolf.keenetic.link
- LETSENCRYPT_HOST=brencher.rudolf.keenetic.link
- PROFILES=brencher2,torrserv_proxy,immich,registry,gmail_mcp
- VERSION=${version}
- PROFILES=brencher2,torrserv_proxy,immich,gmail_mcp,hermes,hermes_infra,gitea,connectivity_tester
- SECONDARY_BRENCHER=http://host.docker.internal:5002
deploy:
replicas: 1
+11
View File
@@ -13,6 +13,7 @@ const closeStatus = document.getElementById('close-status');
const refreshBranchesBtn = document.getElementById('refresh-branches');
const applyChangesBtn = document.getElementById('apply-changes');
const branchFilter = document.getElementById('branch-filter');
const appVersion = document.getElementById('app-version');
// Branches list (flattened) for UI rendering: [{ envId, envName, branch, commits: [] }]
let branches = [];
@@ -477,6 +478,8 @@ function setupWebSockets() {
filterBranches();
renderJobs();
showStatus('Environments updated.');
} else if ('version' in message) {
appVersion.textContent = `Version: ${message.version || 'unknown'}`;
} else if ('error' in message) {
console.error('Error from server:', message.error);
if (message.error && message.error.code === 'BRANCH_STATE_CONFLICT' && message.error.envId) {
@@ -529,6 +532,14 @@ function updateEnvironment() {
showStatus('Changes applied (pending server confirmation).');
}
// Initial load
(function applyUrlFilter() {
const params = new URLSearchParams(window.location.search);
const initialFilter = params.get('filter');
if (initialFilter && branchFilter) {
branchFilter.value = initialFilter;
filterBranches();
}
})();
setupWebSockets();
showStatus('Loading branches...');
renderBranches();
+1
View File
@@ -10,6 +10,7 @@
<div class="container">
<header>
<h1>Brencher - Branch Merger</h1>
<p class="version" id="app-version">Version: loading...</p>
<div id="status-bar" class="status-bar hidden">
<span id="status-message"></span>
<button id="close-status">×</button>
+7
View File
@@ -15,6 +15,13 @@ body {
padding: 32px;
}
.version {
color: #6c757d;
font-family: monospace;
font-size: 0.85rem;
margin-top: -12px;
}
.primary-button {
background: #007bff;
color: #fff;
+4
View File
@@ -11,3 +11,7 @@ types-PyYAML
types-requests
types-docker
testcontainers
+116
View File
@@ -0,0 +1,116 @@
from typing import Any
import pytest
from enironment import AbstractStep, Environment
from steps.actions import GiteaActionsAwait, GiteaActionsRun, GiteaActionsTrigger
from steps.shared_state import SharedStateHolderInMemory
class FixedRun(AbstractStep[GiteaActionsRun]):
def __init__(self) -> None:
super().__init__()
self.calls = 0
def progress(self) -> GiteaActionsRun:
self.calls += 1
return GiteaActionsRun(42, "queued", None, "http://example/run/42")
def _environment(*steps: AbstractStep[Any], dry: bool) -> Environment:
return Environment(
id="actions-test",
state=SharedStateHolderInMemory(unmerge=None),
pipeline=list(steps),
)
def _trigger() -> GiteaActionsTrigger:
return GiteaActionsTrigger(
api_base="https://gitea.example/api/v1",
owner="owner",
repo="repo",
workflow_file="release.yml",
ref="main",
token="token",
)
def test_trigger_skips_dispatch_in_dry_run() -> None:
trigger = _trigger()
env = _environment(trigger, dry=True)
env.state.set_dry(True)
trigger._http = lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("HTTP call")) # type: ignore[method-assign]
result = trigger.progress()
assert result.status == "dry-run"
assert result.run_id == 0
def test_trigger_reuses_active_run(monkeypatch: pytest.MonkeyPatch) -> None:
trigger = _trigger()
env = _environment(trigger, dry=False)
runs = iter([
None,
GiteaActionsRun(42, "in_progress", None, "http://example/run/42"),
GiteaActionsRun(42, "in_progress", None, "http://example/run/42"),
])
dispatches = 0
def fake_http(method: str, url: str, body: dict[str, Any] | None = None) -> dict[str, Any]:
nonlocal dispatches
if method == "POST":
dispatches += 1
return {}
monkeypatch.setattr(trigger, "_list_recent_runs", lambda: next(runs))
monkeypatch.setattr(trigger, "_http", fake_http)
trigger.poll_interval_sec = 0
trigger.poll_timeout_sec = 1
first = trigger.progress()
second = trigger.progress()
assert first.run_id == 42
assert second.run_id == 42
assert dispatches == 1
def test_await_skips_polling_in_dry_run() -> None:
run = FixedRun()
await_step = GiteaActionsAwait(
api_base="https://gitea.example/api/v1",
owner="owner",
repo="repo",
token="token",
run=run,
)
env = _environment(run, await_step, dry=True)
env.state.set_dry(True)
await_step._http = lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("HTTP call")) # type: ignore[method-assign]
result = await_step.progress()
assert result.status == "dry-run"
assert run.calls == 0
def test_await_raises_when_run_is_not_completed(monkeypatch: pytest.MonkeyPatch) -> None:
run = FixedRun()
await_step = GiteaActionsAwait(
api_base="https://gitea.example/api/v1",
owner="owner",
repo="repo",
token="token",
run=run,
)
env = _environment(run, await_step, dry=False)
monkeypatch.setattr(
await_step,
"_http",
lambda *args, **kwargs: {"status": "in_progress", "conclusion": None},
)
with pytest.raises(RuntimeError, match="is not completed"):
await_step.progress()
+86 -1
View File
@@ -5,10 +5,12 @@ These tests run in a Docker-like isolated environment to avoid harming the host
They simulate remote and local git repositories and test various merge scenarios.
"""
import os
import shutil
import tempfile
import git
import pytest
from steps.git import CheckoutAndMergeResult, GitUnmergeResult
from steps.git import CheckoutAndMergeResult, GitClone, GitUnmergeResult
from .test_remote_repo import RemoteRepoHelper
@@ -16,6 +18,89 @@ from .test_remote_repo import RemoteRepoHelper
class TestGitIntegration:
"""Integration tests for git operations"""
def test_git_clone_multiple_urls(self, repo_helper: RemoteRepoHelper) -> None:
repo_helper.create_commit(repo_helper.repo, "master", "master", "README.md", "content", "Initial commit")
second_url = f"file://{repo_helper.remote_dir}"
clone = GitClone(url=[repo_helper.remote_dir, second_url])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
assert {remote.name for remote in repo.remotes} == {"origin", "repo1"}
assert repo.remotes.origin.url == repo_helper.remote_dir
assert repo.remotes.repo1.url == second_url
assert set(clone.get_branches()) == {"master", "repo1/master"}
def test_git_clone_named_remotes(self, repo_helper: RemoteRepoHelper) -> None:
# Two real local repos served as separate "remotes". The user controls
# their names explicitly via the `remotes=[{name,url}]` API.
def fresh_repo(prefix: str, message: str) -> str:
path = tempfile.mkdtemp(prefix=prefix)
repo = git.Repo.init(path, bare=False)
with repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
repo_helper.create_commit(repo, "master", "master", "README.md", "content", message)
return path
github_dir = fresh_repo("test_named_github_", "github initial")
gitea_dir = fresh_repo("test_named_gitea_", "gitea initial")
try:
clone = GitClone(remotes=[
{"name": "github", "url": github_dir},
{"name": "gitea_admin", "url": gitea_dir},
])
clone.env = repo_helper.env
repo_path = clone.progress()
repo = git.Repo(repo_path)
# User-chosen names appear exactly; the auto-generated "origin" /
# "repo1" names from the positional-list form do NOT leak in.
assert {r.name for r in repo.remotes} == {"github", "gitea_admin"}
assert "origin" not in repo.remotes
assert "repo1" not in repo.remotes
assert repo.remotes.github.url == github_dir
assert repo.remotes.gitea_admin.url == gitea_dir
# `get_branches` keeps the <remote>/<branch> shape for non-origin
# names; the user gets the full provenance in the key.
assert set(clone.get_branches()) == {"github/master", "gitea_admin/master"}
finally:
shutil.rmtree(github_dir, ignore_errors=True)
shutil.rmtree(gitea_dir, ignore_errors=True)
def test_git_clone_remotes_validation(self) -> None:
# Empty list: rejected (would produce an unusable clone).
with pytest.raises(ValueError, match="At least one repository URL is required"):
GitClone(remotes=[])
# Missing / empty / non-string name or url: rejected.
with pytest.raises(ValueError, match="non-empty 'name'"):
GitClone(remotes=[{"url": "git@example.com:foo/bar.git"}])
with pytest.raises(ValueError, match="non-empty 'name'"):
GitClone(remotes=[{"name": "", "url": "git@example.com:foo/bar.git"}])
with pytest.raises(ValueError, match="non-empty 'url'"):
GitClone(remotes=[{"name": "github"}])
with pytest.raises(ValueError, match="non-empty 'url'"):
GitClone(remotes=[{"name": "github", "url": ""}])
# Duplicate name: rejected.
with pytest.raises(ValueError, match="Duplicate remote name"):
GitClone(remotes=[
{"name": "github", "url": "git@example.com:a/b.git"},
{"name": "github", "url": "git@example.com:c/d.git"},
])
# Passing both `url` and `remotes`: rejected.
with pytest.raises(ValueError, match="not both"):
GitClone(url="git@example.com:a/b.git", remotes=[{"name": "github", "url": "git@example.com:a/b.git"}])
# Passing neither: rejected.
with pytest.raises(ValueError, match="requires either 'url' or 'remotes'"):
GitClone()
def test_checkout_merged_one_branch(self, repo_helper: RemoteRepoHelper) -> None:
# Create main branch with initial commit
commit1 = repo_helper.create_commit(repo_helper.repo, "master", "master", "file1.txt", "content1",