20 Commits
Author SHA1 Message Date
hermes e381f46b4d fix(k3s): support GoProxy on armv7 nodes (#19)
## Problem
The GoProxy egress-check init container only handled `x86_64` and `aarch64`. On the `xu4` node, `uname -m` is `armv7l`, so the init container exited with `unsupported architecture: armv7l` and the Deployment stayed in CrashLoopBackOff.

## Fix
- select the upstream GoProxy v15.3 `proxy-linux-arm-v7.tar.gz` asset for `armv7l`;
- verify it with its pinned SHA-256 checksum;
- document ARMv7 support.

This PR is limited to the GoProxy workload and does not change the registry-DNS or connectivity-tester PRs.

Validation: `git diff --check`, architecture-to-asset mapping for `x86_64`, `aarch64`, and `armv7l`, and SHA-256 verification of the downloaded upstream ARMv7 release asset.
Reviewed-on: org1/swarm_connectivity_tester#19
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-21 18:59:20 +01:00
hermes 1cdfe25d32 feat(k3s): add GoProxy egress check POC (#17)
## Summary

- add a Kubernetes GoProxy egress-check POC using `snail007/goproxy`
- keep the existing Squid egress-check workload
- check `http://example.com/` and `http://example.com/robots.txt` separately
- route the client HTTP traffic through the GoProxy sidecar

## Verification

- PyYAML manifest parsing and structural assertions: PASS
- local upstream integration through GoProxy: `/` returned `200`, `/robots.txt` returned `404`
- `git diff --check`: PASS

The Kubernetes cluster was not changed; rollout remains Flux-controlled after merge.

Reviewed-on: org1/swarm_connectivity_tester#17
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-18 23:58:48 +01:00
hermes 1389a3b84c fix(k3s): route checker requests through squid (#16)
Route the checker HTTP requests through the local Squid forward listener while retaining transparent interception.

Reviewed-on: org1/swarm_connectivity_tester#16
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 16:37:37 +01:00
hermes 664db3a784 fix(k3s): put squid forward port first (#15)
Place the regular forward-proxy port before the intercepted port so Squid getMyPort() resolves a valid non-interception listener during startup.

Reviewed-on: org1/swarm_connectivity_tester#15
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 15:59:20 +01:00
hermes 20d43598c9 fix(k3s): add squid forward proxy port (#14)
Keep the transparent interception listener on 3128 and add a regular forward-proxy listener on 3129 so Squid internal URLs have a valid port.

Reviewed-on: org1/swarm_connectivity_tester#14
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 15:18:49 +01:00
hermes aa00e66346 fix(k3s): clear squid init pid before startup (#13)
Run squid -z in foreground mode and remove its PID file before starting the long-running Squid process.

Reviewed-on: org1/swarm_connectivity_tester#13
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 11:27:53 +01:00
hermes 88045795b3 fix(k3s): initialize squid swap directories (#12)
The Squid startup log explicitly requires squid -z before normal startup when using the UFS cache store. Initialize swap directories after creating the top-level cache path.

Reviewed-on: org1/swarm_connectivity_tester#12
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 09:04:52 +01:00
hermes f39d32ad89 fix(k3s): create squid cache directory before startup (#11)
Squid documentation requires the cache_dir top-level directory to exist and be writable; Squid only creates its L1/L2 subdirectories. Create /tmp/squid in the container command before starting Squid with the UFS store.

Reviewed-on: org1/swarm_connectivity_tester#11
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 08:08:04 +01:00
hermes 3d7351b4a7 fix(k3s): use writable squid cache path (#10)
The mounted emptyDir cache path remains unusable for the Squid UFS initializer even with permissive modes. Use /tmp/squid, which is writable in the container, and remove the unnecessary cache volume/initContainer.

Reviewed-on: org1/swarm_connectivity_tester#10
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 00:31:08 +01:00
hermes b215d5bbfb fix(k3s): allow squid cache directory writes (#9)
The init container now completes, but Squid still cannot create nested UFS swap directories with mode 0770. Use mode 0777 on this disposable emptyDir cache volume so the runtime user can initialize its cache store.

Reviewed-on: org1/swarm_connectivity_tester#9
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-15 22:09:14 +01:00
hermes 39604d4c65 fix(k3s): allow squid cache directory initialization (#8)
The init container cannot chown the emptyDir volume: chown returns Operation not permitted. Keep fsGroup 13 and only create the UFS directory plus chmod 0770 so Squid uid 13 can initialize its swap directories.

Reviewed-on: org1/swarm_connectivity_tester#8
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-15 21:46:02 +01:00
hermes 5ac4fe6ea6 fix(k3s): initialize squid cache directories (#7)
Initialize the UFS cache directory as root before starting Squid, then hand ownership to uid/gid 13. Squid requires a supported cache_dir store and the runtime image cannot create its swap directories under the non-root user.

Reviewed-on: org1/swarm_connectivity_tester#7
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-15 20:24:34 +01:00
hermes c5580de386 fix(k3s): use null squid cache store (#6)
Follow-up to #5. Squid still exits while initializing the ufs store because the runtime image cannot create the swap directories. Use the null cache store; caching is already disabled with cache deny all, and the writable mount remains for the configured path.

Reviewed-on: org1/swarm_connectivity_tester#6
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-15 14:00:20 +01:00
hermes 73a1473929 fix(k3s): configure squid cache store (#5)
Follow-up to #4.

The writable /run fix is deployed, but Squid exits because the custom configuration has no cache_dir stores configured. Add a minimal writable ufs store backed by an emptyDir at /var/spool/squid; caching remains disabled with cache deny all.

Reviewed-on: org1/swarm_connectivity_tester#5
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-15 13:45:21 +01:00
hermes 01e838cce3 fix(k3s): make egress checker runtime healthy (#4)
Fix the live egress-check deployment failure.

- provide writable /run for Squid pid creation with fsGroup 13
- remove HTTP probes from the client, which does not run an HTTP server

Observed production failures: Squid exited with permission denied for /run/squid.pid; client was killed by /healthz and /status probes.

Reviewed-on: org1/swarm_connectivity_tester#4
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-15 13:10:19 +01:00
hermesandhermes 12d13f6a7c feat(k3s): add transparent HTTP egress checker (#3)
## Summary
Add a small Kubernetes client that checks two configurable HTTP URLs on the same domain with different paths.

## Changes
- add the `egress-check` namespace, Service, Deployment, client ConfigMap, and Squid ConfigMap
- redirect client HTTP egress to the Squid sidecar with iptables in the shared pod network namespace
- reject direct non-DNS TCP/UDP egress from the pod
- expose `/status` and `/healthz`
- document the HTTP-only boundary; HTTPS transparent proxying requires TLS interception and a trusted CA

## Verification
- `python3 -m py_compile k3s/app.py`
- `git diff --check`
- reviewed the Kustomize resource list and manifest

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: org1/swarm_connectivity_tester#3
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-14 22:29:27 +01:00
admin 9286552737 x 2026-10-16 23:13:36 +02:00
admin 29b036965c x 2026-10-16 19:11:06 +02:00
admin d62b12c67d x 2026-10-16 19:10:10 +02:00
admin e51e40b9e5 x 2026-10-16 19:03:10 +02:00
14 changed files with 479 additions and 225 deletions
+4 -3
View File
@@ -2,6 +2,8 @@ name: Build K3s Connectivity Image
on:
push:
branches:
- main
paths:
- k3s/**
- .gitea/workflows/build-k3s-image.yaml
@@ -22,7 +24,6 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to the Gitea registry
if: gitea.event_name != 'pull_request'
uses: docker/login-action@v3
with:
registry: gitea.rudolf.keenetic.link
@@ -32,8 +33,8 @@ jobs:
uses: docker/build-push-action@v6
with:
context: ./k3s
platforms: linux/amd64,linux/arm64
push: ${{ gitea.event_name != 'pull_request' }}
platforms: linux/amd64,linux/arm64,linux/arm/v7
push: true
tags: |
gitea.rudolf.keenetic.link/org1/swarm_connectivity_tester:latest
gitea.rudolf.keenetic.link/org1/swarm_connectivity_tester:${{ gitea.sha }}
+1 -1
View File
@@ -1 +1 @@
__pycache__/
**/__pycache__/
Binary file not shown.
Binary file not shown.
+20
View File
@@ -0,0 +1,20 @@
# Deploy
```sh
kubectl apply -k k3s
kubectl -n connectivity-test rollout status statefulset/k3s-connectivity
```
# Check
```sh
kubectl -n connectivity-test get pods -o wide
kubectl -n connectivity-test exec k3s-connectivity-0 -- \
python -c "from urllib.request import urlopen; print(urlopen('http://localhost:8080/check').read().decode())"
```
# Undeploy
```sh
kubectl delete -k k3s
```
+48 -97
View File
@@ -1,108 +1,59 @@
# K3s Cross-Node Connectivity Tester
This workload checks HTTP connectivity between StatefulSet pods by their
individual internal DNS names. A successful `/probe` response includes the
source and peer node names and reports `all_cross_node: true` only when every
peer responded from a different node.
## Functional Requirements
The default deployment uses two replicas and HTTP on port `8080`.
- Run one tester pod on each of the three target nodes, with required hostname
anti-affinity preventing more than one tester pod on any node.
- Use parallel StatefulSet replicas with stable internal names:
`k3s-connectivity-0`, `k3s-connectivity-1`, and `k3s-connectivity-2`.
- Probe every other replica over HTTP by its internal Kubernetes DNS name, such
as `k3s-connectivity-1.k3s-connectivity.connectivity-test.svc.cluster.local`.
- Expose `/healthz`, `/identity`, and `/check` on port `8080`.
- Keep `/healthz` local and fast; it must not perform DNS or peer requests.
- Run peer checks only when `/check` is requested and report source/peer pod and
node names.
- Report `all_reachable: true` only when every peer responds successfully.
- Report `all_cross_node: true` only when every successful peer response comes
from a different node.
- Support HTTPS through the mounted `k3s-connectivity-tls` Secret.
- Pull the multi-architecture registry image for `amd64`, `arm64`, and
`arm/v7` nodes.
- Leave pods Pending rather than co-locating them when insufficient eligible
nodes are available; nodes with a disk-pressure `NoSchedule` taint are not
eligible.
## Build and distribute the image
## Transparent egress check
The included StatefulSet pulls:
`egress-check.yaml` adds a client and Squid sidecar in one pod. The client
checks `CHECK_URL_1` and `CHECK_URL_2` every 30 seconds; the URLs must use HTTP,
share a domain, and have different paths. The init container installs iptables
rules in the shared pod network namespace. Client HTTP traffic is redirected
to Squid, while direct TCP and UDP traffic other than DNS is rejected.
```text
gitea.rudolf.keenetic.link/org1/swarm_connectivity_tester:latest
```
This is transparent HTTP proxying. HTTPS cannot be transparently proxied by
this configuration because it would require TLS interception and a trusted CA.
Use an explicit HTTPS proxy or add a separately managed TLS interception design
if HTTPS endpoints are required.
To build a replacement image locally:
The client prints each URL and returned status to the container log, then repeats
both requests every 30 seconds. Change the two URL environment variables in the
Deployment before applying it.
```sh
docker build -t k3s-connectivity-tester:0.1.0 .
```
## GoProxy egress check
The Gitea workflow publishes both `linux/amd64` and `linux/arm64` variants. In
the Gitea repository's Actions secrets, configure:
`goproxy-egress-check.yaml` adds a second client/proxy POC in the
`egress-check-goproxy` namespace. The client checks the same two URLs every 30
seconds and uses the GoProxy sidecar at `127.0.0.1:3128` through the HTTP proxy
environment variables. The sidecar uses the multi-architecture `alpine:3.22`
image and an init container downloads the matching pinned GoProxy v15.3 binary
for `amd64`, `arm64`, or `armv7` with SHA-256 verification.
- `REGISTRY_USERNAME`: a Gitea user or bot account with package write access.
- `REGISTRY_TOKEN`: that account's access token with package write access.
The init container selects the release asset from the runtime architecture, so
this workload is not tied to a single Kubernetes node platform. GoProxy runs as
`proxy http -t tcp -p :3128`.
Push the workflow update, wait for the `Build K3s Connectivity Image` action to
finish, then restart the StatefulSet to pull the multi-architecture image:
```sh
kubectl -n connectivity-test rollout restart statefulset/k3s-connectivity
kubectl -n connectivity-test rollout status statefulset/k3s-connectivity
```
For a multi-node K3s cluster, the image registry must be reachable from every
node. Alternatively, import a local image into the K3s containerd image store
on every node:
```sh
docker save k3s-connectivity-tester:0.1.0 | sudo k3s ctr images import -
```
## Deploy and verify HTTP
```sh
kubectl apply -k .
kubectl -n connectivity-test rollout status statefulset/k3s-connectivity
kubectl -n connectivity-test get pods -o wide
```
The two pods must show different values in the `NODE` column. This is enforced
by required pod anti-affinity and a `DoNotSchedule` hostname spread rule. On a
single-node cluster, the second pod deliberately remains Pending rather than
silently testing same-node traffic.
Nodes carrying the `node.kubernetes.io/disk-pressure:NoSchedule` taint are not
used. Resolve their disk pressure before expecting the second pod to schedule;
do not remove that taint just to run this diagnostic workload.
Run an immediate probe from the first pod:
```sh
kubectl -n connectivity-test exec k3s-connectivity-0 -- \
python -c "from urllib.request import urlopen; print(urlopen('http://localhost:8080/probe').read().decode())"
```
The response contains a peer URL such as:
```text
http://k3s-connectivity-1.k3s-connectivity.connectivity-test.svc.cluster.local:8080/healthz
```
Check the latest periodic result without triggering another request:
```sh
kubectl -n connectivity-test exec k3s-connectivity-0 -- \
python -c "from urllib.request import urlopen; print(urlopen('http://localhost:8080/results').read().decode())"
```
`all_reachable` and `all_cross_node` must both be `true`. The response also
includes the exact source and peer node names, which makes failed placement or
network-policy diagnosis straightforward.
## Enable HTTPS
Create a TLS Secret in the workload namespace, then apply the included
strategic merge patch:
```sh
kubectl -n connectivity-test create secret tls k3s-connectivity-tls \
--cert=tls.crt --key=tls.key
kubectl -n connectivity-test patch statefulset k3s-connectivity \
--type=strategic --patch-file=https-patch.yaml
kubectl -n connectivity-test rollout status statefulset/k3s-connectivity
```
The HTTPS patch disables certificate verification for a self-signed diagnostic
certificate. For a certificate issued by a trusted internal CA with SANs for
the pod DNS names, set `VERIFY_TLS` to `true` in the StatefulSet.
To remove the test:
```sh
kubectl delete -k .
```
This variant uses explicit HTTP proxying rather than iptables interception. The
client has no proxy bypass list, and the proxy and client run in the same pod so
each request is sent to GoProxy before leaving the pod. GoProxy logs are
available with `kubectl logs -n egress-check-goproxy deploy/egress-check-goproxy
-c goproxy`; client results are available from the `client` container.
Binary file not shown.
Binary file not shown.
+50 -118
View File
@@ -1,15 +1,12 @@
#!/usr/bin/env python3
"""HTTP/HTTPS connectivity probe for StatefulSet pods."""
"""Simple HTTP endpoint and on-demand peer connectivity checker."""
from __future__ import annotations
import argparse
import json
import logging
import os
import socket
import ssl
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
@@ -20,21 +17,20 @@ from urllib.parse import urlsplit
from urllib.request import Request, urlopen
LOG = logging.getLogger("k3s-connectivity")
CLUSTER_DOMAIN = "cluster.local"
TLS_CERT_FILE = "/tls/tls.crt"
TLS_KEY_FILE = "/tls/tls.key"
@dataclass(frozen=True)
class Settings:
app_name: str
namespace: str
cluster_domain: str
replicas: int
protocol: str
port: int
request_timeout_seconds: float
probe_interval_seconds: float
verify_tls: bool
tls_cert_file: str
tls_key_file: str
@classmethod
def from_env(cls, environment: Mapping[str, str] | None = None) -> "Settings":
@@ -46,31 +42,25 @@ class Settings:
replicas = int(environment.get("REPLICAS", "2"))
port = int(environment.get("PORT", "8080"))
timeout = float(environment.get("REQUEST_TIMEOUT_SECONDS", "5"))
interval = float(environment.get("PROBE_INTERVAL_SECONDS", "30"))
if replicas < 1 or port < 1 or timeout <= 0 or interval < 0:
raise ValueError("invalid replicas, port, timeout, or probe interval")
if replicas < 1 or port < 1 or timeout <= 0:
raise ValueError("invalid replicas, port, or request timeout")
return cls(
app_name=environment.get("APP_NAME", "k3s-connectivity"),
namespace=environment.get("POD_NAMESPACE", "default"),
cluster_domain=environment.get("CLUSTER_DOMAIN", "cluster.local"),
replicas=replicas,
protocol=protocol,
port=port,
request_timeout_seconds=timeout,
probe_interval_seconds=interval,
verify_tls=environment.get("VERIFY_TLS", "false").lower()
in {"1", "true", "yes"},
tls_cert_file=environment.get("TLS_CERT_FILE", "/tls/tls.crt"),
tls_key_file=environment.get("TLS_KEY_FILE", "/tls/tls.key"),
)
def peer_targets(self, hostname: str) -> list["PeerTarget"]:
local_name = hostname.split(".", maxsplit=1)[0]
suffix = f"{self.app_name}.{self.namespace}.svc.{self.cluster_domain}"
suffix = f"{self.app_name}.{self.namespace}.svc.{CLUSTER_DOMAIN}"
return [
PeerTarget(
ordinal=ordinal,
name=f"{self.app_name}-{ordinal}",
host=f"{self.app_name}-{ordinal}.{suffix}",
)
@@ -81,7 +71,6 @@ class Settings:
@dataclass(frozen=True)
class PeerTarget:
ordinal: int
name: str
host: str
@@ -101,20 +90,6 @@ class PodIdentity:
)
class ProbeStore:
def __init__(self) -> None:
self._lock = threading.Lock()
self._latest: dict[str, Any] | None = None
def replace(self, report: dict[str, Any]) -> None:
with self._lock:
self._latest = report
def latest(self) -> dict[str, Any]:
with self._lock:
return self._latest or {"state": "not-run"}
def _timestamp() -> str:
return datetime.now(timezone.utc).isoformat()
@@ -127,41 +102,35 @@ def _ssl_context(settings: Settings) -> ssl.SSLContext | None:
def probe_peer(target: PeerTarget, settings: Settings,
source: PodIdentity) -> dict[str, Any]:
url = f"{settings.protocol}://{target.host}:{settings.port}/healthz"
checked_at = _timestamp()
url = f"{settings.protocol}://{target.host}:{settings.port}/identity"
request = Request(url, headers={"Accept": "application/json"})
try:
with urlopen(request, timeout=settings.request_timeout_seconds,
context=_ssl_context(settings)) as response:
payload = json.loads(response.read().decode("utf-8"))
peer = payload.get("pod", {})
peer_node = peer.get("node_name", "")
return {
"target": target.name,
"url": url,
"checked_at": checked_at,
"success": 200 <= response.status < 300,
"status": response.status,
"peer_hostname": peer.get("hostname", ""),
"peer_node": peer_node,
"cross_node": bool(source.node_name and peer_node)
and source.node_name != peer_node,
"detail": "ok",
}
peer = payload.get("pod", {})
peer_node = peer.get("node_name", "")
return {
"target": target.name,
"url": url,
"reachable": 200 <= response.status < 300,
"peer_hostname": peer.get("hostname", ""),
"peer_node": peer_node,
"cross_node": bool(source.node_name and peer_node)
and source.node_name != peer_node,
"detail": "ok",
}
except HTTPError as error:
return _failed_result(target, url, checked_at, f"HTTP {error.code}")
return _failed_result(target, url, f"HTTP {error.code}")
except (URLError, OSError, TimeoutError, ValueError) as error:
return _failed_result(target, url, checked_at, str(error))
return _failed_result(target, url, str(error))
def _failed_result(target: PeerTarget, url: str, checked_at: str,
detail: str) -> dict[str, Any]:
def _failed_result(target: PeerTarget, url: str, detail: str) -> dict[str, Any]:
return {
"target": target.name,
"url": url,
"checked_at": checked_at,
"success": False,
"status": None,
"reachable": False,
"peer_hostname": "",
"peer_node": "",
"cross_node": False,
@@ -169,52 +138,43 @@ def _failed_result(target: PeerTarget, url: str, checked_at: str,
}
class ProbeApplication:
class ConnectivityApplication:
def __init__(self, settings: Settings, identity: PodIdentity) -> None:
self.settings = settings
self.identity = identity
self.store = ProbeStore()
def health(self) -> dict[str, Any]:
return {"status": "ok", "pod": asdict(self.identity)}
def health(self) -> dict[str, str]:
return {"status": "ok"}
def probe_now(self) -> dict[str, Any]:
targets = self.settings.peer_targets(self.identity.hostname)
results: list[dict[str, Any]] = []
if targets:
with ThreadPoolExecutor(max_workers=min(len(targets), 8)) as executor:
futures = {
executor.submit(probe_peer, target, self.settings, self.identity): target
for target in targets
}
for future in as_completed(futures):
results.append(future.result())
results.sort(key=lambda result: result["target"])
report = {
def identity_response(self) -> dict[str, dict[str, str]]:
return {"pod": asdict(self.identity)}
def check_peers(self) -> dict[str, Any]:
peers = [
probe_peer(target, self.settings, self.identity)
for target in self.settings.peer_targets(self.identity.hostname)
]
return {
"checked_at": _timestamp(),
"source": asdict(self.identity),
"protocol": self.settings.protocol,
"all_reachable": all(result["success"] for result in results),
"all_cross_node": all(result["cross_node"] for result in results),
"peers": results,
"all_reachable": all(peer["reachable"] for peer in peers),
"all_cross_node": all(peer["cross_node"] for peer in peers),
"peers": peers,
}
self.store.replace(report)
return report
class ConnectivityRequestHandler(BaseHTTPRequestHandler):
server: ThreadingHTTPServer
def do_GET(self) -> None:
application: ProbeApplication = getattr(self.server, "application")
application: ConnectivityApplication = getattr(self.server, "application")
path = urlsplit(self.path).path
if path in {"/", "/healthz"}:
self._send_json(200, application.health())
elif path == "/probe":
report = application.probe_now()
self._send_json(200 if report["all_reachable"] else 503, report)
elif path == "/results":
self._send_json(200, application.store.latest())
elif path == "/identity":
self._send_json(200, application.identity_response())
elif path == "/check":
self._send_json(200, application.check_peers())
else:
self._send_json(404, {"error": "not found"})
@@ -237,49 +197,22 @@ class ConnectivityRequestHandler(BaseHTTPRequestHandler):
def _configure_tls(server: ThreadingHTTPServer, settings: Settings) -> None:
if settings.protocol != "https":
return
if not Path(settings.tls_cert_file).is_file() or not Path(settings.tls_key_file).is_file():
raise FileNotFoundError("TLS_CERT_FILE and TLS_KEY_FILE must reference mounted files")
if not Path(TLS_CERT_FILE).is_file() or not Path(TLS_KEY_FILE).is_file():
raise FileNotFoundError("mounted TLS certificate and key are missing")
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain(settings.tls_cert_file, settings.tls_key_file)
context.load_cert_chain(TLS_CERT_FILE, TLS_KEY_FILE)
server.socket = context.wrap_socket(server.socket, server_side=True)
def _run_periodic_probes(application: ProbeApplication,
stop_event: threading.Event) -> None:
while not stop_event.is_set():
report = application.probe_now()
LOG.info("probe complete reachable=%s cross_node=%s peers=%d",
report["all_reachable"], report["all_cross_node"],
len(report["peers"]))
if application.settings.probe_interval_seconds == 0:
return
stop_event.wait(application.settings.probe_interval_seconds)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--once", action="store_true")
arguments = parser.parse_args()
logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"),
logging.basicConfig(level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s")
settings = Settings.from_env()
application = ProbeApplication(settings, PodIdentity.local())
if arguments.once:
report = application.probe_now()
print(json.dumps(report, sort_keys=True))
return 0 if report["all_reachable"] else 1
application = ConnectivityApplication(settings, PodIdentity.local())
server = ThreadingHTTPServer(("", settings.port), ConnectivityRequestHandler)
server.daemon_threads = True
setattr(server, "application", application)
_configure_tls(server, settings)
stop_event = threading.Event()
probe_thread = threading.Thread(
target=_run_periodic_probes,
args=(application, stop_event),
daemon=True,
)
probe_thread.start()
try:
LOG.info("listening protocol=%s port=%d pod=%s node=%s",
settings.protocol, settings.port, application.identity.hostname,
@@ -288,7 +221,6 @@ def main() -> int:
except KeyboardInterrupt:
return 0
finally:
stop_event.set()
server.shutdown()
server.server_close()
+147
View File
@@ -0,0 +1,147 @@
apiVersion: v1
kind: Namespace
metadata:
name: egress-check
---
apiVersion: v1
kind: ConfigMap
metadata:
name: egress-check-client
namespace: egress-check
data:
client.py: |
import os
import time
from urllib.request import urlopen
urls = [os.environ["CHECK_URL_1"], os.environ["CHECK_URL_2"]]
timeout = float(os.environ.get("CHECK_TIMEOUT_SECONDS", "5"))
interval = float(os.environ.get("CHECK_INTERVAL_SECONDS", "30"))
while True:
for url in urls:
try:
with urlopen(url, timeout=timeout) as response:
print(url, response.status, flush=True)
except Exception as error:
print(url, error, flush=True)
time.sleep(interval)
---
apiVersion: v1
kind: ConfigMap
metadata:
name: egress-check-squid
namespace: egress-check
data:
squid.conf: |
http_port 3129
http_port 3128 intercept
acl localnet src 0.0.0.0/0
http_access allow localnet
http_access deny all
cache_dir ufs /tmp/squid 1 1 1
cache deny all
access_log stdio:/var/log/squid/access.log
shutdown_lifetime 1 seconds
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: egress-check
namespace: egress-check
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: egress-check
template:
metadata:
labels:
app.kubernetes.io/name: egress-check
spec:
automountServiceAccountToken: false
securityContext:
fsGroup: 13
initContainers:
- name: redirect-egress
image: alpine:3.20
command:
- /bin/sh
- -ec
- apk add --no-cache iptables && iptables -t nat -A OUTPUT -o lo -j ACCEPT && iptables -t nat -A OUTPUT -m owner --uid-owner 13 -j ACCEPT && iptables -t nat -A OUTPUT -p tcp --dport 80 -j REDIRECT --to-ports 3128 && iptables -A OUTPUT -o lo -j ACCEPT && iptables -A OUTPUT -m owner --uid-owner 13 -j ACCEPT && iptables -A OUTPUT -p udp --dport 53 -j ACCEPT && iptables -A OUTPUT -p tcp --dport 53 -j ACCEPT && iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT && iptables -A OUTPUT -p tcp -j REJECT && iptables -A OUTPUT -p udp -j REJECT
securityContext:
runAsUser: 0
capabilities:
add:
- NET_ADMIN
- NET_RAW
drop:
- ALL
containers:
- name: client
image: python:3.12-alpine
command:
- python
- /app/client.py
env:
- name: CHECK_URL_1
value: http://example.com/
- name: CHECK_URL_2
value: http://example.com/robots.txt
- name: CHECK_INTERVAL_SECONDS
value: "30"
- name: CHECK_TIMEOUT_SECONDS
value: "5"
- name: http_proxy
value: http://127.0.0.1:3129
- name: https_proxy
value: http://127.0.0.1:3129
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: client
mountPath: /app
readOnly: true
- name: squid
image: ubuntu/squid:latest
command:
- /bin/sh
- -ec
- mkdir -p /tmp/squid && chmod 0777 /tmp/squid && /usr/sbin/squid -z -N -f /etc/squid/squid.conf && rm -f /run/squid.pid && exec /usr/sbin/squid -N -d 1
ports:
- name: proxy
containerPort: 3128
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
runAsUser: 13
runAsGroup: 13
volumeMounts:
- name: squid
mountPath: /etc/squid/squid.conf
subPath: squid.conf
readOnly: true
- name: squid-run
mountPath: /run
volumes:
- name: client
configMap:
name: egress-check-client
- name: squid
configMap:
name: egress-check-squid
- name: squid-run
emptyDir: {}
+161
View File
@@ -0,0 +1,161 @@
apiVersion: v1
kind: Namespace
metadata:
name: egress-check-goproxy
---
apiVersion: v1
kind: ConfigMap
metadata:
name: egress-check-goproxy-client
namespace: egress-check-goproxy
data:
client.py: |
import os
import time
from urllib.request import urlopen
urls = [os.environ["CHECK_URL_1"], os.environ["CHECK_URL_2"]]
timeout = float(os.environ.get("CHECK_TIMEOUT_SECONDS", "5"))
interval = float(os.environ.get("CHECK_INTERVAL_SECONDS", "30"))
while True:
for url in urls:
try:
with urlopen(url, timeout=timeout) as response:
print(url, response.status, flush=True)
except Exception as error:
print(url, error, flush=True)
time.sleep(interval)
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: egress-check-goproxy
namespace: egress-check-goproxy
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: egress-check-goproxy
template:
metadata:
labels:
app.kubernetes.io/name: egress-check-goproxy
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile:
type: RuntimeDefault
initContainers:
- name: goproxy-download
image: alpine:3.22@sha256:49340b346798875b5592e5d420241f2f9cfba5f47e62ebb1446a1b34734e898f
command:
- /bin/sh
- -ec
- |
case "$(uname -m)" in
x86_64)
asset=proxy-linux-amd64.tar.gz
checksum=00be86a17b16c4444fe86766e715c86a95d9e8e7bdf1b0b902ecd8bcf9cd8a54
;;
aarch64)
asset=proxy-linux-arm64-v8.tar.gz
checksum=dd230006c13caee98d66233a2b5764ad8d3d58e6ea081cec61344ada8f55d015
;;
armv7l)
asset=proxy-linux-arm-v7.tar.gz
checksum=78190e0c0ab4e979654a373d4677619e06ed962ba46bc0367398f92b010319c0
;;
*)
echo "unsupported architecture: $(uname -m)" >&2
exit 1
;;
esac
wget -qO /tmp/proxy.tar.gz "https://github.com/snail007/goproxy/releases/download/v15.3/$asset"
echo "$checksum /tmp/proxy.tar.gz" | sha256sum -c -
tar -xzf /tmp/proxy.tar.gz -C /proxy
chmod 0555 /proxy/proxy
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
volumeMounts:
- name: goproxy
mountPath: /proxy
- name: tmp
mountPath: /tmp
containers:
- name: client
image: python:3.12-alpine
command:
- python
- /app/client.py
env:
- name: CHECK_URL_1
value: http://example.com/
- name: CHECK_URL_2
value: http://example.com/robots.txt
- name: CHECK_INTERVAL_SECONDS
value: "30"
- name: CHECK_TIMEOUT_SECONDS
value: "5"
- name: http_proxy
value: http://127.0.0.1:3128
- name: https_proxy
value: http://127.0.0.1:3128
- name: HTTP_PROXY
value: http://127.0.0.1:3128
- name: HTTPS_PROXY
value: http://127.0.0.1:3128
- name: no_proxy
value: ""
- name: NO_PROXY
value: ""
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
volumeMounts:
- name: client
mountPath: /app
readOnly: true
- name: goproxy
image: alpine:3.22@sha256:49340b346798875b5592e5d420241f2f9cfba5f47e62ebb1446a1b34734e898f
command:
- /proxy/proxy
- http
- -t
- tcp
- -p
- :3128
ports:
- name: proxy
containerPort: 3128
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
volumeMounts:
- name: goproxy
mountPath: /proxy
readOnly: true
- name: tmp
mountPath: /tmp
volumes:
- name: goproxy
emptyDir: {}
- name: client
configMap:
name: egress-check-goproxy-client
- name: tmp
emptyDir: {}
+3 -4
View File
@@ -30,7 +30,7 @@ metadata:
app.kubernetes.io/name: k3s-connectivity
spec:
serviceName: k3s-connectivity
replicas: 2
replicas: 3
podManagementPolicy: Parallel
selector:
matchLabels:
@@ -80,13 +80,11 @@ spec:
fieldRef:
fieldPath: status.podIP
- name: REPLICAS
value: "2"
value: "3"
- name: PROTOCOL
value: http
- name: PORT
value: "8080"
- name: PROBE_INTERVAL_SECONDS
value: "30"
- name: REQUEST_TIMEOUT_SECONDS
value: "5"
readinessProbe:
@@ -113,6 +111,7 @@ spec:
port: web
scheme: HTTP
periodSeconds: 2
timeoutSeconds: 2
failureThreshold: 15
resources:
requests:
+3 -1
View File
@@ -1,4 +1,6 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kubernetes.yaml
- kubernetes.yaml
- egress-check.yaml
- goproxy-egress-check.yaml
+42 -1
View File
@@ -1,6 +1,11 @@
import json
import threading
import unittest
from http.server import ThreadingHTTPServer
from urllib.request import urlopen
from app import Settings
from app import (ConnectivityApplication, ConnectivityRequestHandler,
PodIdentity, Settings)
class SettingsTests(unittest.TestCase):
@@ -27,5 +32,41 @@ class SettingsTests(unittest.TestCase):
Settings.from_env({"PROTOCOL": "tcp"})
class EndpointTests(unittest.TestCase):
def setUp(self) -> None:
settings = Settings.from_env({"REPLICAS": "1", "PORT": "8080"})
application = ConnectivityApplication(
settings,
PodIdentity("k3s-connectivity-0", "test-node", "127.0.0.1"),
)
self.server = ThreadingHTTPServer(
("127.0.0.1", 0), ConnectivityRequestHandler)
self.server.daemon_threads = True
setattr(self.server, "application", application)
self.thread = threading.Thread(target=self.server.serve_forever)
self.thread.start()
self.base_url = f"http://127.0.0.1:{self.server.server_port}"
def tearDown(self) -> None:
self.server.shutdown()
self.server.server_close()
self.thread.join()
def test_health_is_local_and_fast(self) -> None:
with urlopen(f"{self.base_url}/healthz", timeout=1) as response:
payload = json.loads(response.read())
self.assertEqual(response.status, 200)
self.assertEqual(payload, {"status": "ok"})
def test_check_succeeds_without_peers(self) -> None:
with urlopen(f"{self.base_url}/check", timeout=1) as response:
payload = json.loads(response.read())
self.assertTrue(payload["all_reachable"])
self.assertTrue(payload["all_cross_node"])
self.assertEqual(payload["peers"], [])
if __name__ == "__main__":
unittest.main()