New Swarm stack connectivity_tester — a long-running Python service that
probes the cluster's connectivity every 10 minutes and logs structured
PASS/FAIL per check.
What's in the box
connectivity_tester/Dockerfile — python:3.12-slim base
connectivity_tester/app.py — main loop: read env, run round, sleep, repeat
Stack connectivity_tester shows up in docker stack ls
docker service ps connectivity_tester_connectivity-tester shows one task per node (odroid, rudolf-T, xu4)
Logs show first round within 30 seconds of start
docker service logs --tail 50 connectivity_tester_connectivity-tester shows PASS for at least squid:3128 and hermes-gateway:8000
Negative-list (169.254.169.254) is unreachable (PASS OK-NEG)
## Summary
New Swarm stack `connectivity_tester` — a long-running Python service that
probes the cluster's connectivity every 10 minutes and logs structured
PASS/FAIL per check.
## What's in the box
- `connectivity_tester/Dockerfile` — python:3.12-slim base
- `connectivity_tester/app.py` — main loop: read env, run round, sleep, repeat
- `connectivity_tester/endpoints.py` — `EXPECTED_INTERNAL`, `EXPECTED_EXTERNAL`, `NEGATIVE` lists
- `connectivity_tester/stack-compose.yml` — Swarm stack with `mode: global` (one task per node)
## Behaviour
- One round every `PERIOD_SECONDS` (default 600).
- Per round: TCP probes against the three allow-lists + the negative-list.
- Negative-list endpoints (cloud metadata services etc.) MUST NOT respond.
If they do → log `ERROR LEAK`, container keeps running.
- `EXTRA_ENDPOINTS` env var accepts comma-separated `host:port` pairs for ad-hoc checks.
## Placement
`mode: global` — Swarm schedules one task on each node. No node pinning.
Networks: `hermes_proxy` overlay (for internal DNS resolution) + `default`
(for egress via squid).
## Acceptance
- After deploy, `docker service logs connectivity_tester_connectivity-tester`
shows one round per 10 minutes.
- Logs visible at info level in normal conditions (squid, hermes-gateway,
etc. all PASS).
- Any unexpected reachability or unreachable internal endpoint triggers
ERROR-level log within `CHECK_TIMEOUT_S`.
## Related
- Org1/hermes-voice-gateway#3 (Path B: build custom voice-gateway) — blocked by Org1/hermes-voice-gateway#5 (connectivity testing container)
- Org1/hermes-voice-gateway#5 — this PR resolves the blocker
## Verification checklist
- [ ] Stack `connectivity_tester` shows up in `docker stack ls`
- [ ] `docker service ps connectivity_tester_connectivity-tester` shows one task per node (odroid, rudolf-T, xu4)
- [ ] Logs show first round within 30 seconds of start
- [ ] `docker service logs --tail 50 connectivity_tester_connectivity-tester` shows PASS for at least squid:3128 and hermes-gateway:8000
- [ ] Negative-list (169.254.169.254) is unreachable (PASS OK-NEG)
Long-running Swarm service that verifies expected endpoints are
reachable from inside the cluster and that the negative-list endpoints
(cloud metadata etc.) stay blocked.
Runs one round every PERIOD_SECONDS (default 600): TCP probes against
EXPECTED_INTERNAL (overlay services), EXPECTED_EXTERNAL (internet
hosts), EXTRA_ENDPOINTS (ad-hoc from env), and NEGATIVE (must NOT
respond; alerts on any leak). Logs structured output, alerts on
failures, container keeps running.
Mode: global — Swarm schedules one task on every node that meets
constraints. Attached to hermes_proxy overlay (for internal DNS) and
default (for egress via squid).
Blocks Org1/hermes-voice-gateway#3 (Path B) until we verify the
overlay + egress paths voice-gateway will depend on.
brencher parses compose via re.sub(r'\$\{([^}]+)\}', lambda m: env[m.group(1)]).
With shell default syntax ${VAR:-default} the regex captures 'VAR:-default'
as a single key, env['VAR:-default'] raises KeyError.
Use bare ${EXTRA_ENDPOINTS}; if the env var is missing brencher's
.env lookup (or the dict.get fallback in the deploy step) handles it.
If EXTRA_ENDPOINTS isn't set at deploy time the value is empty, which
the python service treats as 'no extras'.
brencher DockerComposeBuild raises KeyError: 'EXTRA_ENDPOINTS' when the
env var is not passed via envs=. Hardcode empty default in compose for
deterministic build.
Add google.com and lan gateways to EXPECTED_EXTERNAL so connectivity-tester
probes them on every round.
Previous 'dockerfile: connectivity_tester/Dockerfile' resolved through
brencher's join to <repo>/connectivity_tester/connectivity_tester/Dockerfile,
which does not exist; build failed with 'Cannot locate specified Dockerfile'.
The Dockerfile sits next to stack-compose.yml at <repo>/connectivity_tester/Dockerfile.
After brencher fix uses context=<repo>/connectivity_tester/ (single-level
now that compose 'dockerfile' is 'Dockerfile'), the previous COPY line
referenced a non-existent connectivity_tester/ subdirectory inside the
build context.
docker stack deploy on this engine rejects 'Additional property name is not allowed'.
Sibling stacks (gitea, hermes, immich) ship without it; the stack name comes from
the brencher 'stack_name' CLI argument instead.
Self-attach/detach to overlay networks via docker socket. At any moment the
container is on exactly one network; per-network list of expected services is
loaded from network_map.NETWORK_MAP. After checks run cross-validation:
- service reachable but not in map -> ERROR 'LEAK-UNEXPECTED'
- service in map but unreachable -> ERROR 'MISSING-EXPECTED'
- global service not in any map -> WARNING 'GLOBAL-UNCOVERED'
Compose uses /var/run/docker.sock bind mount and removes overlay-network
references; EXTRA_ENDPOINTS / HTTP_PROXY / HTTPS_PROXY / NO_PROXY are gone.
endpoints.py removed (its data moved into network_map.py).
Replace tuple-of-strings with frozen Endpoint dataclass carrying optional
(proxy_host, proxy_port) tuple. Add network_map.NETWORK_MAP populated from
the live swarm endpoints discovered via the docker socket, and a separate
PUBLIC_SERVICES list sourced from the squid allowlist plus well-known
public hosts. Endpoints with proxy are checked via HTTP CONNECT through
that proxy; endpoints without are probed directly via TCP after attaching
to the relevant overlay network. Cross-validation rules cover leaks and
missing expectations per round.
Aligns with the proven hermes/reverseproxy@main build.yml: uses
secrets.REGISTRY_USERNAME/REGISTRY_PASSWORD (which already exist in the
cluster-wide Actions runner scope), docker/setup-buildx-action@v3 and
docker/build-push-action@v6 from docker/github-mirror, and targets the
gitea.rudolf.keenetic.link/<image>:auto-<sha> plus :latest tags.
Multi-arch matrix now covers linux/amd64, linux/arm64 and linux/arm/v7
so push to Gitea Container Registry will fan out to rudolf-T (x86_64),
odroid (aarch64) and xu4 (32-bit ARM).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
New Swarm stack
connectivity_tester— a long-running Python service thatprobes the cluster's connectivity every 10 minutes and logs structured
PASS/FAIL per check.
What's in the box
connectivity_tester/Dockerfile— python:3.12-slim baseconnectivity_tester/app.py— main loop: read env, run round, sleep, repeatconnectivity_tester/endpoints.py—EXPECTED_INTERNAL,EXPECTED_EXTERNAL,NEGATIVElistsconnectivity_tester/stack-compose.yml— Swarm stack withmode: global(one task per node)Behaviour
PERIOD_SECONDS(default 600).If they do → log
ERROR LEAK, container keeps running.EXTRA_ENDPOINTSenv var accepts comma-separatedhost:portpairs for ad-hoc checks.Placement
mode: global— Swarm schedules one task on each node. No node pinning.Networks:
hermes_proxyoverlay (for internal DNS resolution) +default(for egress via squid).
Acceptance
docker service logs connectivity_tester_connectivity-testershows one round per 10 minutes.
etc. all PASS).
ERROR-level log within
CHECK_TIMEOUT_S.Related
Verification checklist
connectivity_testershows up indocker stack lsdocker service ps connectivity_tester_connectivity-testershows one task per node (odroid, rudolf-T, xu4)docker service logs --tail 50 connectivity_tester_connectivity-testershows PASS for at least squid:3128 and hermes-gateway:8000hermes referenced this pull request from admin/brencher2026-09-14 17:47:18 +02:00
brencher parses compose via re.sub(r'\$\{([^}]+)\}', lambda m: env[m.group(1)]). With shell default syntax ${VAR:-default} the regex captures 'VAR:-default' as a single key, env['VAR:-default'] raises KeyError. Use bare ${EXTRA_ENDPOINTS}; if the env var is missing brencher's .env lookup (or the dict.get fallback in the deploy step) handles it. If EXTRA_ENDPOINTS isn't set at deploy time the value is empty, which the python service treats as 'no extras'.