feat(connectivity_tester): new stack for synthetic connectivity checks #39

Merged
admin merged 12 commits from hermes/ansible:feat/connectivity-tester into master 2026-09-20 13:26:10 +02:00
Contributor

Summary

New Swarm stack connectivity_tester — a long-running Python service that
probes the cluster's connectivity every 10 minutes and logs structured
PASS/FAIL per check.

What's in the box

  • connectivity_tester/Dockerfile — python:3.12-slim base
  • connectivity_tester/app.py — main loop: read env, run round, sleep, repeat
  • connectivity_tester/endpoints.py — EXPECTED_INTERNAL, EXPECTED_EXTERNAL, NEGATIVE lists
  • connectivity_tester/stack-compose.yml — Swarm stack with mode: global (one task per node)

Behaviour

  • One round every PERIOD_SECONDS (default 600).
  • Per round: TCP probes against the three allow-lists + the negative-list.
  • Negative-list endpoints (cloud metadata services etc.) MUST NOT respond.
    If they do → log ERROR LEAK, container keeps running.
  • EXTRA_ENDPOINTS env var accepts comma-separated host:port pairs for ad-hoc checks.

Placement

mode: global — Swarm schedules one task on each node. No node pinning.

Networks: hermes_proxy overlay (for internal DNS resolution) + default
(for egress via squid).

Acceptance

  • After deploy, docker service logs connectivity_tester_connectivity-tester
    shows one round per 10 minutes.
  • Logs visible at info level in normal conditions (squid, hermes-gateway,
    etc. all PASS).
  • Any unexpected reachability or unreachable internal endpoint triggers
    ERROR-level log within CHECK_TIMEOUT_S.

Related

Verification checklist

  • Stack connectivity_tester shows up in docker stack ls
  • docker service ps connectivity_tester_connectivity-tester shows one task per node (odroid, rudolf-T, xu4)
  • Logs show first round within 30 seconds of start
  • docker service logs --tail 50 connectivity_tester_connectivity-tester shows PASS for at least squid:3128 and hermes-gateway:8000
  • Negative-list (169.254.169.254) is unreachable (PASS OK-NEG)
## Summary New Swarm stack `connectivity_tester` — a long-running Python service that probes the cluster's connectivity every 10 minutes and logs structured PASS/FAIL per check. ## What's in the box - `connectivity_tester/Dockerfile` — python:3.12-slim base - `connectivity_tester/app.py` — main loop: read env, run round, sleep, repeat - `connectivity_tester/endpoints.py` — `EXPECTED_INTERNAL`, `EXPECTED_EXTERNAL`, `NEGATIVE` lists - `connectivity_tester/stack-compose.yml` — Swarm stack with `mode: global` (one task per node) ## Behaviour - One round every `PERIOD_SECONDS` (default 600). - Per round: TCP probes against the three allow-lists + the negative-list. - Negative-list endpoints (cloud metadata services etc.) MUST NOT respond. If they do → log `ERROR LEAK`, container keeps running. - `EXTRA_ENDPOINTS` env var accepts comma-separated `host:port` pairs for ad-hoc checks. ## Placement `mode: global` — Swarm schedules one task on each node. No node pinning. Networks: `hermes_proxy` overlay (for internal DNS resolution) + `default` (for egress via squid). ## Acceptance - After deploy, `docker service logs connectivity_tester_connectivity-tester` shows one round per 10 minutes. - Logs visible at info level in normal conditions (squid, hermes-gateway, etc. all PASS). - Any unexpected reachability or unreachable internal endpoint triggers ERROR-level log within `CHECK_TIMEOUT_S`. ## Related - Org1/hermes-voice-gateway#3 (Path B: build custom voice-gateway) — blocked by Org1/hermes-voice-gateway#5 (connectivity testing container) - Org1/hermes-voice-gateway#5 — this PR resolves the blocker ## Verification checklist - [ ] Stack `connectivity_tester` shows up in `docker stack ls` - [ ] `docker service ps connectivity_tester_connectivity-tester` shows one task per node (odroid, rudolf-T, xu4) - [ ] Logs show first round within 30 seconds of start - [ ] `docker service logs --tail 50 connectivity_tester_connectivity-tester` shows PASS for at least squid:3128 and hermes-gateway:8000 - [ ] Negative-list (169.254.169.254) is unreachable (PASS OK-NEG)
hermes added 1 commit 2026-09-14 11:40:15 +02:00
Long-running Swarm service that verifies expected endpoints are
reachable from inside the cluster and that the negative-list endpoints
(cloud metadata etc.) stay blocked.

Runs one round every PERIOD_SECONDS (default 600): TCP probes against
EXPECTED_INTERNAL (overlay services), EXPECTED_EXTERNAL (internet
hosts), EXTRA_ENDPOINTS (ad-hoc from env), and NEGATIVE (must NOT
respond; alerts on any leak). Logs structured output, alerts on
failures, container keeps running.

Mode: global — Swarm schedules one task on every node that meets
constraints. Attached to hermes_proxy overlay (for internal DNS) and
default (for egress via squid).

Blocks Org1/hermes-voice-gateway#3 (Path B) until we verify the
overlay + egress paths voice-gateway will depend on.
hermes added 1 commit 2026-09-14 21:27:00 +02:00
brencher parses compose via re.sub(r'\$\{([^}]+)\}', lambda m: env[m.group(1)]).
With shell default syntax ${VAR:-default} the regex captures 'VAR:-default'
as a single key, env['VAR:-default'] raises KeyError.

Use bare ${EXTRA_ENDPOINTS}; if the env var is missing brencher's
.env lookup (or the dict.get fallback in the deploy step) handles it.
If EXTRA_ENDPOINTS isn't set at deploy time the value is empty, which
the python service treats as 'no extras'.
hermes added 1 commit 2026-09-15 07:04:19 +02:00
brencher DockerComposeBuild raises KeyError: 'EXTRA_ENDPOINTS' when the
env var is not passed via envs=. Hardcode empty default in compose for
deterministic build.

Add google.com and lan gateways to EXPECTED_EXTERNAL so connectivity-tester
probes them on every round.
hermes added 1 commit 2026-09-15 09:39:49 +02:00
Previous 'dockerfile: connectivity_tester/Dockerfile' resolved through
brencher's join to <repo>/connectivity_tester/connectivity_tester/Dockerfile,
which does not exist; build failed with 'Cannot locate specified Dockerfile'.
The Dockerfile sits next to stack-compose.yml at <repo>/connectivity_tester/Dockerfile.
hermes added 1 commit 2026-09-15 09:59:13 +02:00
After brencher fix uses context=<repo>/connectivity_tester/ (single-level
now that compose 'dockerfile' is 'Dockerfile'), the previous COPY line
referenced a non-existent connectivity_tester/ subdirectory inside the
build context.
hermes added 1 commit 2026-09-15 10:19:24 +02:00
docker stack deploy on this engine rejects 'Additional property name is not allowed'.
Sibling stacks (gitea, hermes, immich) ship without it; the stack name comes from
the brencher 'stack_name' CLI argument instead.
hermes added 1 commit 2026-09-15 11:24:03 +02:00
Self-attach/detach to overlay networks via docker socket. At any moment the
container is on exactly one network; per-network list of expected services is
loaded from network_map.NETWORK_MAP. After checks run cross-validation:

- service reachable but not in map  -> ERROR 'LEAK-UNEXPECTED'
- service in map but unreachable    -> ERROR 'MISSING-EXPECTED'
- global service not in any map     -> WARNING 'GLOBAL-UNCOVERED'

Compose uses /var/run/docker.sock bind mount and removes overlay-network
references; EXTRA_ENDPOINTS / HTTP_PROXY / HTTPS_PROXY / NO_PROXY are gone.
endpoints.py removed (its data moved into network_map.py).
hermes added 1 commit 2026-09-15 12:11:55 +02:00
Replace tuple-of-strings with frozen Endpoint dataclass carrying optional
(proxy_host, proxy_port) tuple. Add network_map.NETWORK_MAP populated from
the live swarm endpoints discovered via the docker socket, and a separate
PUBLIC_SERVICES list sourced from the squid allowlist plus well-known
public hosts. Endpoints with proxy are checked via HTTP CONNECT through
that proxy; endpoints without are probed directly via TCP after attaching
to the relevant overlay network. Cross-validation rules cover leaks and
missing expectations per round.
hermes added 1 commit 2026-09-15 19:42:53 +02:00
hermes added 1 commit 2026-09-15 21:06:29 +02:00
Aligns with the proven hermes/reverseproxy@main build.yml: uses
secrets.REGISTRY_USERNAME/REGISTRY_PASSWORD (which already exist in the
cluster-wide Actions runner scope), docker/setup-buildx-action@v3 and
docker/build-push-action@v6 from docker/github-mirror, and targets the
gitea.rudolf.keenetic.link/<image>:auto-<sha> plus :latest tags.

Multi-arch matrix now covers linux/amd64, linux/arm64 and linux/arm/v7
so push to Gitea Container Registry will fan out to rudolf-T (x86_64),
odroid (aarch64) and xu4 (32-bit ARM).
hermes added 1 commit 2026-09-15 21:20:57 +02:00
hermes added 1 commit 2026-09-15 22:21:28 +02:00
admin merged commit ff08677d12 into master 2026-09-20 13:26:09 +02:00
admin deleted branch feat/connectivity-tester 2026-09-20 13:26:11 +02:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/ansible#39