Compare commits

...
Author SHA1 Message Date
hermes 607641e56b feat(connectivity): probe swarm host ports and Telegram egress 2026-09-30 13:59:01 +00:00
hermes d17a93626f fix(ci): enable multi-platform buildx driver 2026-09-29 20:24:27 +00:00
hermes 6511ae28a8 fix(connectivity): dispatch multi-arch admin image build 2026-09-29 20:15:33 +00:00
hermes 98e6f251a6 merge admin master into connectivity release 2026-09-29 18:50:28 +00:00
hermes 703e0198cf ci(connectivity_tester): voice-gateway style (single-arch, push tag trigger) 2026-09-18 13:06:41 +00:00
hermes 9af2cd992f fix(ci): use REGISTRY_USERNAME/TOKEN (proven pattern from voice-gateway) 2026-09-18 11:58:00 +00:00
hermes faa3e32a53 Reapply "fix(ci): use auto-injected GITHUB_TOKEN instead of REGISTRY_USERNAME/TOKEN secrets"
This reverts commit e6d184bd95.
2026-09-17 22:09:37 +00:00
hermes e6d184bd95 Revert "fix(ci): use auto-injected GITHUB_TOKEN instead of REGISTRY_USERNAME/TOKEN secrets"
This reverts commit d097862c81.
2026-09-17 21:43:55 +00:00
hermes d097862c81 fix(ci): use auto-injected GITHUB_TOKEN instead of REGISTRY_USERNAME/TOKEN secrets 2026-09-17 21:41:13 +00:00
hermes b2e85b686c fix(ci): sanitize platform in image tag (linux/amd64 -> linux-amd64) 2026-09-17 19:06:07 +00:00
admin 932be82ae6 ci(connectivity_tester): drop build block; image pulled from Gitea CR 2026-09-17 19:27:17 +02:00
admin 953e4dcba7 ci(connectivity_tester): switch stack image to Gitea Container Registry 2026-09-17 18:32:39 +02:00
admin 81e0641ab1 fix(ci): use REGISTRY_USERNAME/TOKEN (which are populated); runner OS via matrix.os 2026-09-17 18:19:15 +02:00
admin 8f71c723fb feat(ci): keep PACKAGE_PUSH_* secrets + multi-arch; login continue-on-error so we see what other side fails 2026-09-17 18:13:47 +02:00
admin 7fed0c08cd feat(ci): dump individual secrets, multi-arch matrix, switch to PACKAGE_PUSH_* 2026-09-17 18:05:58 +02:00
admin 4b8ece95c0 feat(ci): dump env+secrets, multi-arch, switch to PACKAGE_PUSH_* 2026-09-17 18:03:55 +02:00
admin 0734adc229 ci(connectivity_tester): copy release.yml to release/0.1.0-connectivity 2026-09-16 23:18:00 +02:00
hermes 8ecc6f0424 ci: drop broken .github/workflows path 2026-09-15 22:21:23 +02:00
admin 4ab39982d1 fix(connectivity_tester): hardcode REGISTRY_USERNAME (missing secret) 2026-09-15 21:20:50 +02:00
admin b5b94b2eee ci(connectivity_tester): rewrite workflow to use REGISTRY_* secrets + multi-arch
Aligns with the proven hermes/reverseproxy@main build.yml: uses
secrets.REGISTRY_USERNAME/REGISTRY_PASSWORD (which already exist in the
cluster-wide Actions runner scope), docker/setup-buildx-action@v3 and
docker/build-push-action@v6 from docker/github-mirror, and targets the
gitea.rudolf.keenetic.link/<image>:auto-<sha> plus :latest tags.

Multi-arch matrix now covers linux/amd64, linux/arm64 and linux/arm/v7
so push to Gitea Container Registry will fan out to rudolf-T (x86_64),
odroid (aarch64) and xu4 (32-bit ARM).
2026-09-15 21:06:24 +02:00
admin bf6b86302f ci(connectivity_tester): add multi-arch Gitea Actions workflow 2026-09-15 19:42:46 +02:00
admin 1e9d33485d feat(connectivity_tester): Endpoint dataclass, proxy-aware checks, public hosts
Replace tuple-of-strings with frozen Endpoint dataclass carrying optional
(proxy_host, proxy_port) tuple. Add network_map.NETWORK_MAP populated from
the live swarm endpoints discovered via the docker socket, and a separate
PUBLIC_SERVICES list sourced from the squid allowlist plus well-known
public hosts. Endpoints with proxy are checked via HTTP CONNECT through
that proxy; endpoints without are probed directly via TCP after attaching
to the relevant overlay network. Cross-validation rules cover leaks and
missing expectations per round.
2026-09-15 10:11:49 +00:00
admin d5b1a23bcb feat(connectivity_tester): rotate networks per-round, cross-validate map
Self-attach/detach to overlay networks via docker socket. At any moment the
container is on exactly one network; per-network list of expected services is
loaded from network_map.NETWORK_MAP. After checks run cross-validation:

- service reachable but not in map  -> ERROR 'LEAK-UNEXPECTED'
- service in map but unreachable    -> ERROR 'MISSING-EXPECTED'
- global service not in any map     -> WARNING 'GLOBAL-UNCOVERED'

Compose uses /var/run/docker.sock bind mount and removes overlay-network
references; EXTRA_ENDPOINTS / HTTP_PROXY / HTTPS_PROXY / NO_PROXY are gone.
endpoints.py removed (its data moved into network_map.py).
2026-09-15 09:23:55 +00:00
admin e94ea37e85 fix(connectivity_tester): drop root name: key from compose
docker stack deploy on this engine rejects 'Additional property name is not allowed'.
Sibling stacks (gitea, hermes, immich) ship without it; the stack name comes from
the brencher 'stack_name' CLI argument instead.
2026-09-15 08:19:19 +00:00
admin c0d53b709c fix(connectivity_tester): COPY . instead of connectivity_tester/
After brencher fix uses context=<repo>/connectivity_tester/ (single-level
now that compose 'dockerfile' is 'Dockerfile'), the previous COPY line
referenced a non-existent connectivity_tester/ subdirectory inside the
build context.
2026-09-15 07:59:06 +00:00
admin 0366a78a96 fix(connectivity_tester): use Dockerfile (single-level) in stack build
Previous 'dockerfile: connectivity_tester/Dockerfile' resolved through
brencher's join to <repo>/connectivity_tester/connectivity_tester/Dockerfile,
which does not exist; build failed with 'Cannot locate specified Dockerfile'.
The Dockerfile sits next to stack-compose.yml at <repo>/connectivity_tester/Dockerfile.
2026-09-15 07:39:43 +00:00
admin c13e9d2ce7 fix(connectivity_tester): define EXTRA_ENDPOINTS empty + add external hosts
brencher DockerComposeBuild raises KeyError: 'EXTRA_ENDPOINTS' when the
env var is not passed via envs=. Hardcode empty default in compose for
deterministic build.

Add google.com and lan gateways to EXPECTED_EXTERNAL so connectivity-tester
probes them on every round.
2026-09-15 05:04:07 +00:00
hermes 020a5092a0 fix(compose): drop shell default syntax from EXTRA_ENDPOINTS
brencher parses compose via re.sub(r'\$\{([^}]+)\}', lambda m: env[m.group(1)]).
With shell default syntax ${VAR:-default} the regex captures 'VAR:-default'
as a single key, env['VAR:-default'] raises KeyError.

Use bare ${EXTRA_ENDPOINTS}; if the env var is missing brencher's
.env lookup (or the dict.get fallback in the deploy step) handles it.
If EXTRA_ENDPOINTS isn't set at deploy time the value is empty, which
the python service treats as 'no extras'.
2026-09-14 19:26:54 +00:00
hermes 5a36921f3b feat(connectivity_tester): new stack for synthetic connectivity checks
Long-running Swarm service that verifies expected endpoints are
reachable from inside the cluster and that the negative-list endpoints
(cloud metadata etc.) stay blocked.

Runs one round every PERIOD_SECONDS (default 600): TCP probes against
EXPECTED_INTERNAL (overlay services), EXPECTED_EXTERNAL (internet
hosts), EXTRA_ENDPOINTS (ad-hoc from env), and NEGATIVE (must NOT
respond; alerts on any leak). Logs structured output, alerts on
failures, container keeps running.

Mode: global — Swarm schedules one task on every node that meets
constraints. Attached to hermes_proxy overlay (for internal DNS) and
default (for egress via squid).

Blocks Org1/hermes-voice-gateway#3 (Path B) until we verify the
overlay + egress paths voice-gateway will depend on.
2026-09-14 09:39:13 +00:00
6 changed files with 160 additions and 43 deletions
+9 -30
View File
@@ -2,16 +2,12 @@ name: release
on:
push:
branches:
- 'auto-*'
tags:
- 'v*'
workflow_dispatch:
inputs:
tag:
description: 'Image tag (e.g. auto-<sha>)'
required: false
default: latest
required: true
type: string
jobs:
@@ -21,28 +17,12 @@ jobs:
- name: checkout
uses: actions/checkout@v4
- name: resolve metadata
id: metadata
run: |
registry="${GITHUB_SERVER_URL#https://}"
registry="${registry#http://}"
owner="${GITHUB_REPOSITORY%%/*}"
if [ -n "${{ inputs.tag }}" ] && [ "${{ inputs.tag }}" != "latest" ]; then
version="${{ inputs.tag }}"
else
version="${GITHUB_REF_NAME##*/}"
fi
echo "registry=${registry}" >> "$GITHUB_OUTPUT"
echo "owner=${owner}" >> "$GITHUB_OUTPUT"
echo "version=${version}" >> "$GITHUB_OUTPUT"
echo "image=${registry}/${owner}/connectivity_tester:${version}" >> "$GITHUB_OUTPUT"
- name: log into Gitea Packages
env:
REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
run: |
printf '%s' "$REGISTRY_TOKEN" | docker login "${{ steps.metadata.outputs.registry }}" --username "$REGISTRY_USERNAME" --password-stdin
uses: docker/login-action@v3
with:
registry: gitea.rudolf.keenetic.link
username: ${{ secrets.REGISTRY_USERNAME }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: setup buildx
uses: docker/setup-buildx-action@v3
@@ -53,8 +33,7 @@ jobs:
context: connectivity_tester
file: connectivity_tester/Dockerfile
push: true
platforms: linux/arm64,linux/arm/v7
provenance: false
platforms: linux/amd64,linux/arm64,linux/arm/v7
tags: |
${{ steps.metadata.outputs.image }}
${{ steps.metadata.outputs.registry }}/${{ steps.metadata.outputs.owner }}/connectivity_tester:latest
gitea.rudolf.keenetic.link/admin/connectivity_tester:${{ inputs.tag || github.ref_name }}
gitea.rudolf.keenetic.link/admin/connectivity_tester:latest
+34 -9
View File
@@ -23,11 +23,14 @@ from docker.models.containers import Container
from docker.models.networks import Network
from network_map import Endpoint, GLOBAL_SERVICES, NEGATIVE, NETWORK_MAP, PUBLIC_SERVICES
from host_checks import build_host_checks, parse_peers, tcp_probe, udp_probe
LOG = logging.getLogger("connectivity-tester")
FALLBACK_TCP_TIMEOUT_S = 3.0
HTTP_CONNECT_TIMEOUT_S = 5.0
HOST_TCP_PORTS = (2377, 7946, 2049)
HOST_UDP_PORTS = (7946, 4789)
def _socket_path() -> str:
@@ -214,6 +217,27 @@ async def _check_negative(endpoints: List[Endpoint],
return leaks
async def _check_host_connectivity(
peers: List[Tuple[str, str]], timeout: float
) -> None:
checks = build_host_checks(peers, HOST_TCP_PORTS, HOST_UDP_PORTS)
LOG.info("--- host connectivity checks (%d probes) ---", len(checks))
for name, address, protocol, port in checks:
if protocol == "tcp":
ok, detail = await tcp_probe(address, port, timeout)
else:
ok, detail = await udp_probe(address, port, timeout)
label = f"{name} {address}:{port}/{protocol}"
if ok:
LOG.info("PASS-HOST %s %s", label, detail)
else:
LOG.error("FAIL-HOST %s %s", label, detail)
await _check_endpoint(
Endpoint("api.telegram.org", 443),
max(timeout, HTTP_CONNECT_TIMEOUT_S),
)
async def main() -> int:
logging.basicConfig(
level=logging.INFO,
@@ -222,7 +246,9 @@ async def main() -> int:
)
period = int(os.environ.get("PERIOD_SECONDS", "600"))
timeout = float(os.environ.get("CHECK_TIMEOUT_S", "3"))
peers = parse_peers(os.environ.get("SWARM_PEERS", ""))
LOG.info("starting; period=%ds timeout=%.1fs", period, timeout)
LOG.info("configured swarm peers: %d", len(peers))
client = docker.DockerClient(base_url="unix://" + _socket_path())
self_container = _self_container(client)
@@ -247,25 +273,24 @@ async def main() -> int:
]
LOG.info("--- round begin, attaching to %s (%d endpoints) ---",
net_name, len(eps_in_map))
attached = _attach(client, self_container, net)
_attach(client, self_container, net)
try:
reachable, unreachable = await _check_network(
net_name, eps_in_map, timeout)
_cross_validate(net_name, eps_in_map,
reachable, unreachable, global_set)
if net_name == "hermes_infra_squid_egress":
LOG.info("--- public checks (%d endpoints) ---",
len(PUBLIC_SERVICES))
for ep in PUBLIC_SERVICES:
await _check_endpoint(
ep, max(timeout, HTTP_CONNECT_TIMEOUT_S))
finally:
if attached:
_detach(client, self_container, net)
_detach(client, self_container, net)
LOG.info("--- round end on %s ---", net_name)
LOG.info("--- public checks (%d endpoints) ---",
len(PUBLIC_SERVICES))
for ep in PUBLIC_SERVICES:
await _check_endpoint(ep, max(timeout, HTTP_CONNECT_TIMEOUT_S))
LOG.info("--- negative checks (%d endpoints) ---", len(NEGATIVE))
await _check_negative(NEGATIVE, timeout)
await _check_host_connectivity(peers, timeout)
except Exception:
LOG.exception("round crashed")
+71
View File
@@ -0,0 +1,71 @@
from __future__ import annotations
import asyncio
import socket
from dataclasses import dataclass
from typing import Iterable, List, Tuple
@dataclass(frozen=True)
class HostCheck:
name: str
address: str
protocol: str
port: int
def parse_peers(value: str) -> List[Tuple[str, str]]:
peers = []
for item in value.split(","):
item = item.strip()
if "=" not in item:
continue
name, address = (part.strip() for part in item.split("=", 1))
if name and address:
peers.append((name, address))
return peers
def build_host_checks(
peers: Iterable[Tuple[str, str]],
tcp_ports: Iterable[int],
udp_ports: Iterable[int],
) -> List[Tuple[str, str, str, int]]:
checks = []
for name, address in peers:
checks.extend((name, address, "tcp", port) for port in tcp_ports)
checks.extend((name, address, "udp", port) for port in udp_ports)
return checks
async def tcp_probe(address: str, port: int, timeout: float) -> Tuple[bool, str]:
try:
_, writer = await asyncio.wait_for(
asyncio.open_connection(address, port), timeout=timeout
)
writer.close()
try:
await writer.wait_closed()
except Exception:
pass
return True, "tcp connect ok"
except asyncio.TimeoutError:
return False, "timeout"
except OSError as exc:
return False, f"{type(exc).__name__}: {exc}"
async def udp_probe(address: str, port: int, timeout: float) -> Tuple[bool, str]:
loop = asyncio.get_running_loop()
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setblocking(False)
try:
await asyncio.wait_for(loop.sock_connect(sock, (address, port)), timeout)
await asyncio.wait_for(loop.sock_sendall(sock, b"connectivity-tester\n"), timeout)
return True, "udp datagram sent; no handshake available"
except asyncio.TimeoutError:
return False, "timeout"
except OSError as exc:
return False, f"{type(exc).__name__}: {exc}"
finally:
sock.close()
+3 -1
View File
@@ -73,7 +73,9 @@ NETWORK_MAP: Dict[str, List[Endpoint]] = {
"docker_socket_proxy_hermes_proxy": [
Endpoint("hermes-gateway", 8000),
],
"connectivity_tester_default": [],
"connectivity_tester_default": [
Endpoint("connectivity-tester", 8765),
],
"brencher_brencher-net": [
Endpoint("brencher-backend", 5000),
],
+2 -3
View File
@@ -1,11 +1,10 @@
services:
connectivity-tester:
build:
context: .
dockerfile: Dockerfile
image: gitea.rudolf.keenetic.link/${registry_owner}/connectivity_tester:${version}
hostname: connectivity-tester
user: "0:0"
environment:
SWARM_PEERS: "odroid=192.168.1.96,rudolf-T=100.105.201.15,xu4=192.168.1.114"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
healthcheck:
+41
View File
@@ -0,0 +1,41 @@
import unittest
from host_checks import build_host_checks, parse_peers
class HostChecksTests(unittest.TestCase):
def test_parse_peers(self):
self.assertEqual(
parse_peers("odroid=192.168.1.96,rudolf-T=100.105.201.15"),
[("odroid", "192.168.1.96"), ("rudolf-T", "100.105.201.15")],
)
def test_parse_peers_ignores_invalid_entries(self):
self.assertEqual(
parse_peers("odroid=192.168.1.96,broken,rudolf-T="),
[("odroid", "192.168.1.96")],
)
def test_build_host_checks(self):
checks = build_host_checks(
[("odroid", "192.168.1.96"), ("rudolf-T", "100.105.201.15")],
[2377, 7946],
[7946, 4789],
)
self.assertEqual(
checks,
[
("odroid", "192.168.1.96", "tcp", 2377),
("odroid", "192.168.1.96", "tcp", 7946),
("odroid", "192.168.1.96", "udp", 7946),
("odroid", "192.168.1.96", "udp", 4789),
("rudolf-T", "100.105.201.15", "tcp", 2377),
("rudolf-T", "100.105.201.15", "tcp", 7946),
("rudolf-T", "100.105.201.15", "udp", 7946),
("rudolf-T", "100.105.201.15", "udp", 4789),
],
)
if __name__ == "__main__":
unittest.main()