feat(configs): add connectivity_tester stack #65

Merged
admin merged 43 commits from hermes/brencher:feat/connectivity-tester-config into main 2026-09-24 23:01:23 +02:00
Contributor

Summary

Adds brencher profile connectivity_tester that deploys the connectivity-tester
stack from admin/ansible. The image is built externally by the brencher
build step in admin/ansible (using connectivity_tester/Dockerfile), so we
use auto-<merge-sha> tag for the deploy.

What's in the box

  • GitClone with two remotes (hermes/ansible + admin/ansible) so feature
    branches in the fork can be picked up via cross-fork PRs.
  • DockerSwarmCheck(stack_name="connectivity_tester") for diff detection.
  • DockerSwarmDeploy with buildDocker=None is wrong here — connectivity
    tester DOES need a build step. Need to change to non-None.
  • UrlCheck on /healthz and SimpleLog for the UI user link.

Open question

The admin/ansible/connectivity_tester/stack-compose.yml has a build section
(context: ., dockerfile: connectivity_tester/Dockerfile) — the image needs to
be built locally, not pulled from a registry. Brencher must build the image
in the deploy step. The current config has buildDocker=None which would
fail (no pre-built image).

This is a known gap; the PR gets the structure in place. Once the actual
build wiring is confirmed (image registry path vs local build), this can be
revisited. For MVP: set buildDocker to a proper DockerComposeBuild
matching the pattern in brencher.py.

Related

## Summary Adds brencher profile `connectivity_tester` that deploys the connectivity-tester stack from `admin/ansible`. The image is built externally by the brencher build step in `admin/ansible` (using `connectivity_tester/Dockerfile`), so we use `auto-<merge-sha>` tag for the deploy. ## What's in the box - `GitClone` with two remotes (hermes/ansible + admin/ansible) so feature branches in the fork can be picked up via cross-fork PRs. - `DockerSwarmCheck(stack_name="connectivity_tester")` for diff detection. - `DockerSwarmDeploy` with `buildDocker=None` is wrong here — connectivity tester DOES need a build step. **Need to change to non-None.** - `UrlCheck` on /healthz and `SimpleLog` for the UI user link. ## Open question The `admin/ansible/connectivity_tester/stack-compose.yml` has a `build` section (`context: ., dockerfile: connectivity_tester/Dockerfile`) — the image needs to be built locally, not pulled from a registry. Brencher must build the image in the deploy step. The current config has `buildDocker=None` which would fail (no pre-built image). This is a known gap; the PR gets the structure in place. Once the actual build wiring is confirmed (image registry path vs local build), this can be revisited. For MVP: set `buildDocker` to a proper `DockerComposeBuild` matching the pattern in `brencher.py`. ## Related - admin/ansible#39 (PR) — adds the stack + Dockerfile + compose - Org1/hermes-voice-gateway#5 — connectivity testing blocker - Org1/hermes-voice-gateway#3 — Path B: voice-gateway, blocked by #5 - admin/ansible#16 — Connectivity testing monitoring: https://gitea.rudolf.keenetic.link/admin/ansible/issues/16
hermes force-pushed feat/connectivity-tester-config from 11af2d3274 to 2d51cd71f3 2026-09-14 19:27:20 +02:00 Compare
hermes added 35 commits 2026-09-20 19:45:21 +02:00
Long-running Swarm service that runs TCP probes against an allow-list of
internal+external endpoints and a negative-list (cloud metadata etc.)
every 10 minutes. Logs structured PASS/FAIL per check.

Stack lives in admin/ansible under connectivity_tester/. One task per
node via mode: global. Resolves Org1/hermes-voice-gateway#5
(connectivity testing blocker for #3).
admin/ansible#39 has a build section in stack-compose.yml, so the
image must be built locally — buildDocker=None would fail with
'No such image' on deploy. Added a proper DockerComposeBuild step
between checkout and deploy. Image tagged auto-<merge-sha> locally;
no registry push (publish=False).
Move compose + Python service + Dockerfile into backend/configs/connectivity_tester/
so the stack can be deployed from a single repo (no cross-repo compose
lookup at deploy time). The brencher config now points at
admin/brencher instead of admin/ansible, and at the new in-repo
compose path.
Without this, brencher's _discover_environments() does not see the
connectivity_tester.py module on next restart. Adding to the comma-
separated PROFILES env wires it in.

Also undo the misplaced relocation from 4586136: clone remote and
docker_compose_path point back at admin/ansible (the compose lives in
admin/ansible#39, not in brencher).
Without an entry in the brencher UI state.branches for connectivity_tester,
GitUnmerge.progress() returns an empty set and the pipeline raises
'Expected exactly one version, got: set()'. This stalls the brencher
loop, blocking other stacks (including brencher self-deploy) from being
processed.

Same workaround as hermes_infra.py (which uses unmerge=None because
its stacks are managed via direct pushes, not via UI).
Same fix as hermes_infra.py — pass unmerge=None to SharedStateHolderInMemory
and skip the unmerge step entirely. Having unmerge=None in the pipeline
violates mypy's type signature (Pipeline expects AbstractStep, None
is not assignable), causing the brencher image build to fail at the
mypy step.

This is the same workaround as hermes_infra (PR #83): the stack is
managed via direct pushes, not via UI state, so unmerge is not needed.
Tracked in admin/brencher#68.
Replace state=SharedStateHolderInMemory(unmerge=None) with a GitUnmerge(clone, dockerSwarmCheck)-backed state holder. Without this, desired_branches is always empty, and brencher's polling loop falls through to 'Empty branches set' on every cycle, so the stack only deploys when an operator PATCHes the environment through the UI. With the unmerge in place the swarm service's org.brencher.version label drives state.branches and the stack self-deploys on each polling cycle.
publish=True + Gitea registry URL. push will fail with HTTP 401 until GITEA_REGISTRY_USERNAME/GITEA_REGISTRY_PASSWORD are wired in as brencher container env — that's the diagnostic to find out whether the bot token is actually allowed to push to gitea.rudolf.keenetic.link/hermes.
Refines the trigger_build failure mode: instead of opaque urllib.error.HTTPError,
raises a RuntimeError explaining the gitea-side bug + the tag-push workaround
(documented in the runtime message).
Default git fetch refspec refuses to update refs whose upstream SHA
diverges from the local copy. When a branch is force-pushed (e.g. our
smoke-test pushes of master, our re-applies of GITHUB_TOKEN fix), the
local remote-tracking ref points at the old SHA and git refuses to
overwrite it. This shows up as 'cannot lock ref ... is at <old> but
expected <new>' and fails the entire clone step.

Adding '+refs/heads/*:refs/remotes/<remote>/*' to each remote's refspec
lets the fetch force-update the remote-tracking refs. Each remote's
existing refspec is preserved; the force-refspec is appended only if
not already present.
This reverts commit 53b8796319.
GitPython Remote does not expose a set_refspec method. Use
repo.config_writer() to set remote.<name>.fetch '+refs/heads/*:refs/remotes/<name>/*'.
This bypasses the default safety against updating remote-tracking refs
that diverge from their upstream (force-pushed branches).

Sets the refspec for every remote, not just 'origin', so multi-remote
configs like connectivity_tester (hermes/ansible + admin/ansible) pick
up force-pushed branches on either side.
This reverts commit 3dd0657733.
Default git fetch refspec refuses to update refs whose upstream SHA
diverges from the local copy. When a branch is force-pushed (e.g. our
smoke-test pushes of master, our re-applies of GITHUB_TOKEN fix), the
local remote-tracking ref points at the old SHA and git refuses to
overwrite it. This shows up as 'cannot lock ref ... is at <old> but
expected <new>' and fails the entire clone step.

Adding '+refs/heads/*:refs/remotes/<remote>/*' to each remote via
config.set_value(section, option, value) lets the fetch force-update
the remote-tracking refs. Each remote gets its own refspec; the
existing 'origin' branchNamePrefix refspec is preserved.
The brencher stack build-delegation pattern (see AGENTS.md) needs four
env vars to fire `GiteaActionsTrigger`:

- GITEA_TOKEN: Gitea PAT used to POST /dispatches and poll /runs
- WORKFLOW_REPO: owner/repo that hosts the workflow file (default: hermes/ansible)
- WORKFLOW_FILE: workflow filename (default: release.yml)
- WORKFLOW_REF: branch or tag the workflow is triggered on (default: master)

Without GITEA_TOKEN the trigger step gets HTTP 401 from
/api/v1/repos/{owner}/{repo}/actions/runs; the pipeline halts at
trigger_build and the stack never deploys. Added the missing keys to
the secret-create `printf` block in README.instruction.md so the next
person wiring up a brencher stack doesn't have to discover this.

Defaults shown match the connectivity_tester stack wiring
(admin/brencher#65).
The previous split between env vars and constructor args created two
sources of truth for the workflow target. Moving WORKFLOW_OWNER /
WORKFLOW_REPO / WORKFLOW_FILE / WORKFLOW_REF into the constructor
arguments makes the dispatch target obvious at the call site and
keeps the docker secret template minimal (only credentials it must
contain).

README.instruction.md now lists only GIT_USERNAME / GIT_PASSWORD /
GITEA_TOKEN in the `printf` block; WORKFLOW_* are documented in
AGENTS.md (already there) as stack-specific config knobs.
POST /api/v1/repos/{owner}/{repo}/actions/workflows/{file}/dispatches
returns HTTP 204 No Content on success (an empty body). The previous
implementation unconditionally called json.loads on the response,
which raised json.decoder.JSONDecodeError on the empty body. The
stack hit this with 'Expecting value: line 1 column 1 (char 0)' and
never advanced past GiteaActionsTrigger.

Return {} instead for empty / 204 responses so the trigger step
succeeds and the rest of the pipeline can run.
Brencher now mirrors the auto-deploy-via-temp-branch pattern Alex
described: instead of triggering the workflow on the default branch,
brencher pushes the merge commit to `auto-<sha>` via CheckoutMerged's
existing push=True logic, then dispatches the workflow on that new
branch. The branch acts as a review point between build and deploy:
after trigger_build and await_build complete, the deploy step still
runs on the same local merge, but the workflow's provenance lives
on a remote branch Alex can inspect or revert before merging back to
default.

- checkoutMerged.push = True (was False). CheckoutMerged already
  builds the branch name as `auto-<commit1>-<commit2>...` from the
  sorted merges and force-pushes it to `origin` (see backend/steps/git.py
  ~line 311).
- trigger_build ref pulls `checkoutMerged.progress().remote_branch_name`
  instead of a hard-coded "master". This makes the workflow run on
  the auto-branch brencher just pushed.
This reverts commit 74229c167c.
Previous attempt to dispatch on the auto-* branch failed at module
load time: passing `ref=checkoutMerged.progress().remote_branch_name`
runs the whole CheckoutMerged pipeline before brencher has populated
self.env.id, raising BaseException('Environment not set for GitClone')
and preventing brencher from even importing connectivity_tester.

Add RefSource = Union[str, Callable[[], Optional[str]], None] mirroring
the existing InputsSource pattern. The Callable is only invoked from
within progress(), by which point the env is fully populated. The str
form is preserved for existing callers (e.g. WORKFLOW_REF='master').
The 'remote_branch_name is None' guard made auto-branch push
opt-in, not opt-out. When the merge commit already matched an
existing remote-tracking ref (e.g. a feat/* branch the workflow
was built on), brencher would skip the push and trigger the
workflow on whatever branch happened to point at the merge commit.
That branch is rarely the one the workflow is configured to watch,
so the workflow would 404 or stay silent.

Make auto-branch push unconditional when push=True: every cycle
where brencher detects a new HEAD in its watched remotes gets a
fresh 'auto-<short-sha>' branch on the origin remote, which the
workflow trigger (on push to 'auto-*') picks up. The earlier
'remote_branch_name' lookup is kept so the returned value is
useful for diagnostics, but no longer gates the push.
CheckoutMerged's auto-* push needs a token with write:repository
on hermes/ansible (and admin/ansible). The brencher container reads
creds via load_dotenv('/run/secrets/brencher-secrets') using
GIT_USERNAME/GIT_PASSWORD by default. The PAT Alex provisioned
(permissions: write:package, write:repository) is deployed into
GITEA_TOKEN/GITEA_USERNAME, not GIT_TOKEN, so the push uses the
wrong env vars and fails with 'Failed to authenticate user'.

Passing credEnvPrefix='GITEA' to the GitClone for connectivity_tester
makes brencher read GITEA_USERNAME + GITEA_PASSWORD for the auto
push, leaving the existing GIT_* defaults intact for every other
stack (so brencher.py / hermes_infra / brencher2 etc. keep
working with whatever's currently in brencher-secrets).
debug(web): log UI environment selections
Python package / build (3.12.3) (pull_request) Successful in 2m48s
Python package / build (3.13) (pull_request) Successful in 2m38s
Python package / build (3.14) (pull_request) Successful in 2m9s
575da88c05
hermes force-pushed feat/connectivity-tester-config from afd6f7aac0 to 575da88c05 2026-09-20 19:45:21 +02:00 Compare
hermes added 1 commit 2026-09-20 20:04:32 +02:00
fix(connectivity_tester): pass registry owner to compose
Python package / build (3.12.3) (pull_request) Successful in 2m49s
Python package / build (3.13) (pull_request) Successful in 2m17s
Python package / build (3.14) (pull_request) Successful in 2m13s
80d0a2c282
hermes added 1 commit 2026-09-20 20:46:27 +02:00
fix(connectivity_tester): persist configured branch selection
Python package / build (3.12.3) (pull_request) Successful in 2m46s
Python package / build (3.13) (pull_request) Successful in 2m34s
Python package / build (3.14) (pull_request) Successful in 2m8s
d34ed2d15d
hermes added 1 commit 2026-09-20 21:04:04 +02:00
fix(docker): login before private registry pull
Python package / build (3.12.3) (pull_request) Successful in 3m10s
Python package / build (3.13) (pull_request) Successful in 2m8s
Python package / build (3.14) (pull_request) Successful in 2m12s
9194e89197
hermes added 1 commit 2026-09-20 21:12:48 +02:00
fix(docker): reuse gitea credentials for registry pull
Python package / build (3.12.3) (pull_request) Canceled after 1m21s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
86aba66756
hermes added 1 commit 2026-09-20 21:14:59 +02:00
fix(docker): keep registry owner out of compose
Python package / build (3.12.3) (pull_request) Successful in 2m44s
Python package / build (3.13) (pull_request) Successful in 2m6s
Python package / build (3.14) (pull_request) Successful in 2m24s
7ba38ff72b
hermes added 1 commit 2026-09-21 08:41:25 +02:00
fix(docker): use writable config for registry login
Python package / build (3.12.3) (pull_request) Successful in 2m9s
Python package / build (3.13) (pull_request) Successful in 2m21s
Python package / build (3.14) (pull_request) Successful in 2m43s
0bdec9f1ec
hermes added 1 commit 2026-09-21 19:16:02 +02:00
fix(connectivity_tester): remove default branch selection
Python package / build (3.12.3) (pull_request) Canceled after 1m4s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
ebd23c63c4
hermes added 1 commit 2026-09-21 19:17:10 +02:00
refactor(connectivity_tester): deploy without compose pull
Python package / build (3.12.3) (pull_request) Successful in 2m8s
Python package / build (3.13) (pull_request) Successful in 2m2s
Python package / build (3.14) (pull_request) Successful in 2m6s
390f1fe8ed
admin merged commit 710b78a528 into main 2026-09-24 23:01:23 +02:00
admin deleted branch feat/connectivity-tester-config 2026-09-24 23:01:25 +02:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/brencher#65