Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e381f46b4d | ||
|
|
1cdfe25d32 | ||
|
|
1389a3b84c | ||
|
|
664db3a784 | ||
|
|
20d43598c9 | ||
|
|
aa00e66346 | ||
|
|
88045795b3 | ||
|
|
f39d32ad89 | ||
|
|
3d7351b4a7 | ||
|
|
b215d5bbfb | ||
|
|
39604d4c65 | ||
|
|
5ac4fe6ea6 | ||
|
|
c5580de386 | ||
|
|
73a1473929 | ||
|
|
01e838cce3 | ||
|
|
12d13f6a7c | ||
|
|
9286552737 | ||
|
|
29b036965c | ||
|
|
d62b12c67d | ||
|
|
e51e40b9e5 |
@@ -2,6 +2,8 @@ name: Build K3s Connectivity Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
paths:
|
||||
- k3s/**
|
||||
- .gitea/workflows/build-k3s-image.yaml
|
||||
@@ -22,7 +24,6 @@ jobs:
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
- name: Log in to the Gitea registry
|
||||
if: gitea.event_name != 'pull_request'
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: gitea.rudolf.keenetic.link
|
||||
@@ -32,8 +33,8 @@ jobs:
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: ./k3s
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: ${{ gitea.event_name != 'pull_request' }}
|
||||
platforms: linux/amd64,linux/arm64,linux/arm/v7
|
||||
push: true
|
||||
tags: |
|
||||
gitea.rudolf.keenetic.link/org1/swarm_connectivity_tester:latest
|
||||
gitea.rudolf.keenetic.link/org1/swarm_connectivity_tester:${{ gitea.sha }}
|
||||
+1
-1
@@ -1 +1 @@
|
||||
__pycache__/
|
||||
**/__pycache__/
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,20 @@
|
||||
# Deploy
|
||||
|
||||
```sh
|
||||
kubectl apply -k k3s
|
||||
kubectl -n connectivity-test rollout status statefulset/k3s-connectivity
|
||||
```
|
||||
|
||||
# Check
|
||||
|
||||
```sh
|
||||
kubectl -n connectivity-test get pods -o wide
|
||||
kubectl -n connectivity-test exec k3s-connectivity-0 -- \
|
||||
python -c "from urllib.request import urlopen; print(urlopen('http://localhost:8080/check').read().decode())"
|
||||
```
|
||||
|
||||
# Undeploy
|
||||
|
||||
```sh
|
||||
kubectl delete -k k3s
|
||||
```
|
||||
+48
-97
@@ -1,108 +1,59 @@
|
||||
# K3s Cross-Node Connectivity Tester
|
||||
|
||||
This workload checks HTTP connectivity between StatefulSet pods by their
|
||||
individual internal DNS names. A successful `/probe` response includes the
|
||||
source and peer node names and reports `all_cross_node: true` only when every
|
||||
peer responded from a different node.
|
||||
## Functional Requirements
|
||||
|
||||
The default deployment uses two replicas and HTTP on port `8080`.
|
||||
- Run one tester pod on each of the three target nodes, with required hostname
|
||||
anti-affinity preventing more than one tester pod on any node.
|
||||
- Use parallel StatefulSet replicas with stable internal names:
|
||||
`k3s-connectivity-0`, `k3s-connectivity-1`, and `k3s-connectivity-2`.
|
||||
- Probe every other replica over HTTP by its internal Kubernetes DNS name, such
|
||||
as `k3s-connectivity-1.k3s-connectivity.connectivity-test.svc.cluster.local`.
|
||||
- Expose `/healthz`, `/identity`, and `/check` on port `8080`.
|
||||
- Keep `/healthz` local and fast; it must not perform DNS or peer requests.
|
||||
- Run peer checks only when `/check` is requested and report source/peer pod and
|
||||
node names.
|
||||
- Report `all_reachable: true` only when every peer responds successfully.
|
||||
- Report `all_cross_node: true` only when every successful peer response comes
|
||||
from a different node.
|
||||
- Support HTTPS through the mounted `k3s-connectivity-tls` Secret.
|
||||
- Pull the multi-architecture registry image for `amd64`, `arm64`, and
|
||||
`arm/v7` nodes.
|
||||
- Leave pods Pending rather than co-locating them when insufficient eligible
|
||||
nodes are available; nodes with a disk-pressure `NoSchedule` taint are not
|
||||
eligible.
|
||||
|
||||
## Build and distribute the image
|
||||
## Transparent egress check
|
||||
|
||||
The included StatefulSet pulls:
|
||||
`egress-check.yaml` adds a client and Squid sidecar in one pod. The client
|
||||
checks `CHECK_URL_1` and `CHECK_URL_2` every 30 seconds; the URLs must use HTTP,
|
||||
share a domain, and have different paths. The init container installs iptables
|
||||
rules in the shared pod network namespace. Client HTTP traffic is redirected
|
||||
to Squid, while direct TCP and UDP traffic other than DNS is rejected.
|
||||
|
||||
```text
|
||||
gitea.rudolf.keenetic.link/org1/swarm_connectivity_tester:latest
|
||||
```
|
||||
This is transparent HTTP proxying. HTTPS cannot be transparently proxied by
|
||||
this configuration because it would require TLS interception and a trusted CA.
|
||||
Use an explicit HTTPS proxy or add a separately managed TLS interception design
|
||||
if HTTPS endpoints are required.
|
||||
|
||||
To build a replacement image locally:
|
||||
The client prints each URL and returned status to the container log, then repeats
|
||||
both requests every 30 seconds. Change the two URL environment variables in the
|
||||
Deployment before applying it.
|
||||
|
||||
```sh
|
||||
docker build -t k3s-connectivity-tester:0.1.0 .
|
||||
```
|
||||
## GoProxy egress check
|
||||
|
||||
The Gitea workflow publishes both `linux/amd64` and `linux/arm64` variants. In
|
||||
the Gitea repository's Actions secrets, configure:
|
||||
`goproxy-egress-check.yaml` adds a second client/proxy POC in the
|
||||
`egress-check-goproxy` namespace. The client checks the same two URLs every 30
|
||||
seconds and uses the GoProxy sidecar at `127.0.0.1:3128` through the HTTP proxy
|
||||
environment variables. The sidecar uses the multi-architecture `alpine:3.22`
|
||||
image and an init container downloads the matching pinned GoProxy v15.3 binary
|
||||
for `amd64`, `arm64`, or `armv7` with SHA-256 verification.
|
||||
|
||||
- `REGISTRY_USERNAME`: a Gitea user or bot account with package write access.
|
||||
- `REGISTRY_TOKEN`: that account's access token with package write access.
|
||||
The init container selects the release asset from the runtime architecture, so
|
||||
this workload is not tied to a single Kubernetes node platform. GoProxy runs as
|
||||
`proxy http -t tcp -p :3128`.
|
||||
|
||||
Push the workflow update, wait for the `Build K3s Connectivity Image` action to
|
||||
finish, then restart the StatefulSet to pull the multi-architecture image:
|
||||
|
||||
```sh
|
||||
kubectl -n connectivity-test rollout restart statefulset/k3s-connectivity
|
||||
kubectl -n connectivity-test rollout status statefulset/k3s-connectivity
|
||||
```
|
||||
|
||||
For a multi-node K3s cluster, the image registry must be reachable from every
|
||||
node. Alternatively, import a local image into the K3s containerd image store
|
||||
on every node:
|
||||
|
||||
```sh
|
||||
docker save k3s-connectivity-tester:0.1.0 | sudo k3s ctr images import -
|
||||
```
|
||||
|
||||
## Deploy and verify HTTP
|
||||
|
||||
```sh
|
||||
kubectl apply -k .
|
||||
kubectl -n connectivity-test rollout status statefulset/k3s-connectivity
|
||||
kubectl -n connectivity-test get pods -o wide
|
||||
```
|
||||
|
||||
The two pods must show different values in the `NODE` column. This is enforced
|
||||
by required pod anti-affinity and a `DoNotSchedule` hostname spread rule. On a
|
||||
single-node cluster, the second pod deliberately remains Pending rather than
|
||||
silently testing same-node traffic.
|
||||
|
||||
Nodes carrying the `node.kubernetes.io/disk-pressure:NoSchedule` taint are not
|
||||
used. Resolve their disk pressure before expecting the second pod to schedule;
|
||||
do not remove that taint just to run this diagnostic workload.
|
||||
|
||||
Run an immediate probe from the first pod:
|
||||
|
||||
```sh
|
||||
kubectl -n connectivity-test exec k3s-connectivity-0 -- \
|
||||
python -c "from urllib.request import urlopen; print(urlopen('http://localhost:8080/probe').read().decode())"
|
||||
```
|
||||
|
||||
The response contains a peer URL such as:
|
||||
|
||||
```text
|
||||
http://k3s-connectivity-1.k3s-connectivity.connectivity-test.svc.cluster.local:8080/healthz
|
||||
```
|
||||
|
||||
Check the latest periodic result without triggering another request:
|
||||
|
||||
```sh
|
||||
kubectl -n connectivity-test exec k3s-connectivity-0 -- \
|
||||
python -c "from urllib.request import urlopen; print(urlopen('http://localhost:8080/results').read().decode())"
|
||||
```
|
||||
|
||||
`all_reachable` and `all_cross_node` must both be `true`. The response also
|
||||
includes the exact source and peer node names, which makes failed placement or
|
||||
network-policy diagnosis straightforward.
|
||||
|
||||
## Enable HTTPS
|
||||
|
||||
Create a TLS Secret in the workload namespace, then apply the included
|
||||
strategic merge patch:
|
||||
|
||||
```sh
|
||||
kubectl -n connectivity-test create secret tls k3s-connectivity-tls \
|
||||
--cert=tls.crt --key=tls.key
|
||||
kubectl -n connectivity-test patch statefulset k3s-connectivity \
|
||||
--type=strategic --patch-file=https-patch.yaml
|
||||
kubectl -n connectivity-test rollout status statefulset/k3s-connectivity
|
||||
```
|
||||
|
||||
The HTTPS patch disables certificate verification for a self-signed diagnostic
|
||||
certificate. For a certificate issued by a trusted internal CA with SANs for
|
||||
the pod DNS names, set `VERIFY_TLS` to `true` in the StatefulSet.
|
||||
|
||||
To remove the test:
|
||||
|
||||
```sh
|
||||
kubectl delete -k .
|
||||
```
|
||||
This variant uses explicit HTTP proxying rather than iptables interception. The
|
||||
client has no proxy bypass list, and the proxy and client run in the same pod so
|
||||
each request is sent to GoProxy before leaving the pod. GoProxy logs are
|
||||
available with `kubectl logs -n egress-check-goproxy deploy/egress-check-goproxy
|
||||
-c goproxy`; client results are available from the `client` container.
|
||||
|
||||
Binary file not shown.
Binary file not shown.
+50
-118
@@ -1,15 +1,12 @@
|
||||
#!/usr/bin/env python3
|
||||
"""HTTP/HTTPS connectivity probe for StatefulSet pods."""
|
||||
"""Simple HTTP endpoint and on-demand peer connectivity checker."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
import ssl
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from dataclasses import asdict, dataclass
|
||||
from datetime import datetime, timezone
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
@@ -20,21 +17,20 @@ from urllib.parse import urlsplit
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
LOG = logging.getLogger("k3s-connectivity")
|
||||
CLUSTER_DOMAIN = "cluster.local"
|
||||
TLS_CERT_FILE = "/tls/tls.crt"
|
||||
TLS_KEY_FILE = "/tls/tls.key"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Settings:
|
||||
app_name: str
|
||||
namespace: str
|
||||
cluster_domain: str
|
||||
replicas: int
|
||||
protocol: str
|
||||
port: int
|
||||
request_timeout_seconds: float
|
||||
probe_interval_seconds: float
|
||||
verify_tls: bool
|
||||
tls_cert_file: str
|
||||
tls_key_file: str
|
||||
|
||||
@classmethod
|
||||
def from_env(cls, environment: Mapping[str, str] | None = None) -> "Settings":
|
||||
@@ -46,31 +42,25 @@ class Settings:
|
||||
replicas = int(environment.get("REPLICAS", "2"))
|
||||
port = int(environment.get("PORT", "8080"))
|
||||
timeout = float(environment.get("REQUEST_TIMEOUT_SECONDS", "5"))
|
||||
interval = float(environment.get("PROBE_INTERVAL_SECONDS", "30"))
|
||||
if replicas < 1 or port < 1 or timeout <= 0 or interval < 0:
|
||||
raise ValueError("invalid replicas, port, timeout, or probe interval")
|
||||
if replicas < 1 or port < 1 or timeout <= 0:
|
||||
raise ValueError("invalid replicas, port, or request timeout")
|
||||
|
||||
return cls(
|
||||
app_name=environment.get("APP_NAME", "k3s-connectivity"),
|
||||
namespace=environment.get("POD_NAMESPACE", "default"),
|
||||
cluster_domain=environment.get("CLUSTER_DOMAIN", "cluster.local"),
|
||||
replicas=replicas,
|
||||
protocol=protocol,
|
||||
port=port,
|
||||
request_timeout_seconds=timeout,
|
||||
probe_interval_seconds=interval,
|
||||
verify_tls=environment.get("VERIFY_TLS", "false").lower()
|
||||
in {"1", "true", "yes"},
|
||||
tls_cert_file=environment.get("TLS_CERT_FILE", "/tls/tls.crt"),
|
||||
tls_key_file=environment.get("TLS_KEY_FILE", "/tls/tls.key"),
|
||||
)
|
||||
|
||||
def peer_targets(self, hostname: str) -> list["PeerTarget"]:
|
||||
local_name = hostname.split(".", maxsplit=1)[0]
|
||||
suffix = f"{self.app_name}.{self.namespace}.svc.{self.cluster_domain}"
|
||||
suffix = f"{self.app_name}.{self.namespace}.svc.{CLUSTER_DOMAIN}"
|
||||
return [
|
||||
PeerTarget(
|
||||
ordinal=ordinal,
|
||||
name=f"{self.app_name}-{ordinal}",
|
||||
host=f"{self.app_name}-{ordinal}.{suffix}",
|
||||
)
|
||||
@@ -81,7 +71,6 @@ class Settings:
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PeerTarget:
|
||||
ordinal: int
|
||||
name: str
|
||||
host: str
|
||||
|
||||
@@ -101,20 +90,6 @@ class PodIdentity:
|
||||
)
|
||||
|
||||
|
||||
class ProbeStore:
|
||||
def __init__(self) -> None:
|
||||
self._lock = threading.Lock()
|
||||
self._latest: dict[str, Any] | None = None
|
||||
|
||||
def replace(self, report: dict[str, Any]) -> None:
|
||||
with self._lock:
|
||||
self._latest = report
|
||||
|
||||
def latest(self) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
return self._latest or {"state": "not-run"}
|
||||
|
||||
|
||||
def _timestamp() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
@@ -127,41 +102,35 @@ def _ssl_context(settings: Settings) -> ssl.SSLContext | None:
|
||||
|
||||
def probe_peer(target: PeerTarget, settings: Settings,
|
||||
source: PodIdentity) -> dict[str, Any]:
|
||||
url = f"{settings.protocol}://{target.host}:{settings.port}/healthz"
|
||||
checked_at = _timestamp()
|
||||
url = f"{settings.protocol}://{target.host}:{settings.port}/identity"
|
||||
request = Request(url, headers={"Accept": "application/json"})
|
||||
try:
|
||||
with urlopen(request, timeout=settings.request_timeout_seconds,
|
||||
context=_ssl_context(settings)) as response:
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
peer = payload.get("pod", {})
|
||||
peer_node = peer.get("node_name", "")
|
||||
return {
|
||||
"target": target.name,
|
||||
"url": url,
|
||||
"checked_at": checked_at,
|
||||
"success": 200 <= response.status < 300,
|
||||
"status": response.status,
|
||||
"peer_hostname": peer.get("hostname", ""),
|
||||
"peer_node": peer_node,
|
||||
"cross_node": bool(source.node_name and peer_node)
|
||||
and source.node_name != peer_node,
|
||||
"detail": "ok",
|
||||
}
|
||||
peer = payload.get("pod", {})
|
||||
peer_node = peer.get("node_name", "")
|
||||
return {
|
||||
"target": target.name,
|
||||
"url": url,
|
||||
"reachable": 200 <= response.status < 300,
|
||||
"peer_hostname": peer.get("hostname", ""),
|
||||
"peer_node": peer_node,
|
||||
"cross_node": bool(source.node_name and peer_node)
|
||||
and source.node_name != peer_node,
|
||||
"detail": "ok",
|
||||
}
|
||||
except HTTPError as error:
|
||||
return _failed_result(target, url, checked_at, f"HTTP {error.code}")
|
||||
return _failed_result(target, url, f"HTTP {error.code}")
|
||||
except (URLError, OSError, TimeoutError, ValueError) as error:
|
||||
return _failed_result(target, url, checked_at, str(error))
|
||||
return _failed_result(target, url, str(error))
|
||||
|
||||
|
||||
def _failed_result(target: PeerTarget, url: str, checked_at: str,
|
||||
detail: str) -> dict[str, Any]:
|
||||
def _failed_result(target: PeerTarget, url: str, detail: str) -> dict[str, Any]:
|
||||
return {
|
||||
"target": target.name,
|
||||
"url": url,
|
||||
"checked_at": checked_at,
|
||||
"success": False,
|
||||
"status": None,
|
||||
"reachable": False,
|
||||
"peer_hostname": "",
|
||||
"peer_node": "",
|
||||
"cross_node": False,
|
||||
@@ -169,52 +138,43 @@ def _failed_result(target: PeerTarget, url: str, checked_at: str,
|
||||
}
|
||||
|
||||
|
||||
class ProbeApplication:
|
||||
class ConnectivityApplication:
|
||||
def __init__(self, settings: Settings, identity: PodIdentity) -> None:
|
||||
self.settings = settings
|
||||
self.identity = identity
|
||||
self.store = ProbeStore()
|
||||
|
||||
def health(self) -> dict[str, Any]:
|
||||
return {"status": "ok", "pod": asdict(self.identity)}
|
||||
def health(self) -> dict[str, str]:
|
||||
return {"status": "ok"}
|
||||
|
||||
def probe_now(self) -> dict[str, Any]:
|
||||
targets = self.settings.peer_targets(self.identity.hostname)
|
||||
results: list[dict[str, Any]] = []
|
||||
if targets:
|
||||
with ThreadPoolExecutor(max_workers=min(len(targets), 8)) as executor:
|
||||
futures = {
|
||||
executor.submit(probe_peer, target, self.settings, self.identity): target
|
||||
for target in targets
|
||||
}
|
||||
for future in as_completed(futures):
|
||||
results.append(future.result())
|
||||
results.sort(key=lambda result: result["target"])
|
||||
report = {
|
||||
def identity_response(self) -> dict[str, dict[str, str]]:
|
||||
return {"pod": asdict(self.identity)}
|
||||
|
||||
def check_peers(self) -> dict[str, Any]:
|
||||
peers = [
|
||||
probe_peer(target, self.settings, self.identity)
|
||||
for target in self.settings.peer_targets(self.identity.hostname)
|
||||
]
|
||||
return {
|
||||
"checked_at": _timestamp(),
|
||||
"source": asdict(self.identity),
|
||||
"protocol": self.settings.protocol,
|
||||
"all_reachable": all(result["success"] for result in results),
|
||||
"all_cross_node": all(result["cross_node"] for result in results),
|
||||
"peers": results,
|
||||
"all_reachable": all(peer["reachable"] for peer in peers),
|
||||
"all_cross_node": all(peer["cross_node"] for peer in peers),
|
||||
"peers": peers,
|
||||
}
|
||||
self.store.replace(report)
|
||||
return report
|
||||
|
||||
|
||||
class ConnectivityRequestHandler(BaseHTTPRequestHandler):
|
||||
server: ThreadingHTTPServer
|
||||
|
||||
def do_GET(self) -> None:
|
||||
application: ProbeApplication = getattr(self.server, "application")
|
||||
application: ConnectivityApplication = getattr(self.server, "application")
|
||||
path = urlsplit(self.path).path
|
||||
if path in {"/", "/healthz"}:
|
||||
self._send_json(200, application.health())
|
||||
elif path == "/probe":
|
||||
report = application.probe_now()
|
||||
self._send_json(200 if report["all_reachable"] else 503, report)
|
||||
elif path == "/results":
|
||||
self._send_json(200, application.store.latest())
|
||||
elif path == "/identity":
|
||||
self._send_json(200, application.identity_response())
|
||||
elif path == "/check":
|
||||
self._send_json(200, application.check_peers())
|
||||
else:
|
||||
self._send_json(404, {"error": "not found"})
|
||||
|
||||
@@ -237,49 +197,22 @@ class ConnectivityRequestHandler(BaseHTTPRequestHandler):
|
||||
def _configure_tls(server: ThreadingHTTPServer, settings: Settings) -> None:
|
||||
if settings.protocol != "https":
|
||||
return
|
||||
if not Path(settings.tls_cert_file).is_file() or not Path(settings.tls_key_file).is_file():
|
||||
raise FileNotFoundError("TLS_CERT_FILE and TLS_KEY_FILE must reference mounted files")
|
||||
if not Path(TLS_CERT_FILE).is_file() or not Path(TLS_KEY_FILE).is_file():
|
||||
raise FileNotFoundError("mounted TLS certificate and key are missing")
|
||||
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
|
||||
context.load_cert_chain(settings.tls_cert_file, settings.tls_key_file)
|
||||
context.load_cert_chain(TLS_CERT_FILE, TLS_KEY_FILE)
|
||||
server.socket = context.wrap_socket(server.socket, server_side=True)
|
||||
|
||||
|
||||
def _run_periodic_probes(application: ProbeApplication,
|
||||
stop_event: threading.Event) -> None:
|
||||
while not stop_event.is_set():
|
||||
report = application.probe_now()
|
||||
LOG.info("probe complete reachable=%s cross_node=%s peers=%d",
|
||||
report["all_reachable"], report["all_cross_node"],
|
||||
len(report["peers"]))
|
||||
if application.settings.probe_interval_seconds == 0:
|
||||
return
|
||||
stop_event.wait(application.settings.probe_interval_seconds)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--once", action="store_true")
|
||||
arguments = parser.parse_args()
|
||||
|
||||
logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"),
|
||||
logging.basicConfig(level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s")
|
||||
settings = Settings.from_env()
|
||||
application = ProbeApplication(settings, PodIdentity.local())
|
||||
if arguments.once:
|
||||
report = application.probe_now()
|
||||
print(json.dumps(report, sort_keys=True))
|
||||
return 0 if report["all_reachable"] else 1
|
||||
|
||||
application = ConnectivityApplication(settings, PodIdentity.local())
|
||||
server = ThreadingHTTPServer(("", settings.port), ConnectivityRequestHandler)
|
||||
server.daemon_threads = True
|
||||
setattr(server, "application", application)
|
||||
_configure_tls(server, settings)
|
||||
stop_event = threading.Event()
|
||||
probe_thread = threading.Thread(
|
||||
target=_run_periodic_probes,
|
||||
args=(application, stop_event),
|
||||
daemon=True,
|
||||
)
|
||||
probe_thread.start()
|
||||
try:
|
||||
LOG.info("listening protocol=%s port=%d pod=%s node=%s",
|
||||
settings.protocol, settings.port, application.identity.hostname,
|
||||
@@ -288,7 +221,6 @@ def main() -> int:
|
||||
except KeyboardInterrupt:
|
||||
return 0
|
||||
finally:
|
||||
stop_event.set()
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: egress-check
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: egress-check-client
|
||||
namespace: egress-check
|
||||
data:
|
||||
client.py: |
|
||||
import os
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
urls = [os.environ["CHECK_URL_1"], os.environ["CHECK_URL_2"]]
|
||||
timeout = float(os.environ.get("CHECK_TIMEOUT_SECONDS", "5"))
|
||||
interval = float(os.environ.get("CHECK_INTERVAL_SECONDS", "30"))
|
||||
while True:
|
||||
for url in urls:
|
||||
try:
|
||||
with urlopen(url, timeout=timeout) as response:
|
||||
print(url, response.status, flush=True)
|
||||
except Exception as error:
|
||||
print(url, error, flush=True)
|
||||
time.sleep(interval)
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: egress-check-squid
|
||||
namespace: egress-check
|
||||
data:
|
||||
squid.conf: |
|
||||
http_port 3129
|
||||
http_port 3128 intercept
|
||||
acl localnet src 0.0.0.0/0
|
||||
http_access allow localnet
|
||||
http_access deny all
|
||||
cache_dir ufs /tmp/squid 1 1 1
|
||||
cache deny all
|
||||
access_log stdio:/var/log/squid/access.log
|
||||
shutdown_lifetime 1 seconds
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: egress-check
|
||||
namespace: egress-check
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: egress-check
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: egress-check
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
fsGroup: 13
|
||||
initContainers:
|
||||
- name: redirect-egress
|
||||
image: alpine:3.20
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- apk add --no-cache iptables && iptables -t nat -A OUTPUT -o lo -j ACCEPT && iptables -t nat -A OUTPUT -m owner --uid-owner 13 -j ACCEPT && iptables -t nat -A OUTPUT -p tcp --dport 80 -j REDIRECT --to-ports 3128 && iptables -A OUTPUT -o lo -j ACCEPT && iptables -A OUTPUT -m owner --uid-owner 13 -j ACCEPT && iptables -A OUTPUT -p udp --dport 53 -j ACCEPT && iptables -A OUTPUT -p tcp --dport 53 -j ACCEPT && iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT && iptables -A OUTPUT -p tcp -j REJECT && iptables -A OUTPUT -p udp -j REJECT
|
||||
securityContext:
|
||||
runAsUser: 0
|
||||
capabilities:
|
||||
add:
|
||||
- NET_ADMIN
|
||||
- NET_RAW
|
||||
drop:
|
||||
- ALL
|
||||
containers:
|
||||
- name: client
|
||||
image: python:3.12-alpine
|
||||
command:
|
||||
- python
|
||||
- /app/client.py
|
||||
env:
|
||||
- name: CHECK_URL_1
|
||||
value: http://example.com/
|
||||
- name: CHECK_URL_2
|
||||
value: http://example.com/robots.txt
|
||||
- name: CHECK_INTERVAL_SECONDS
|
||||
value: "30"
|
||||
- name: CHECK_TIMEOUT_SECONDS
|
||||
value: "5"
|
||||
- name: http_proxy
|
||||
value: http://127.0.0.1:3129
|
||||
- name: https_proxy
|
||||
value: http://127.0.0.1:3129
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumeMounts:
|
||||
- name: client
|
||||
mountPath: /app
|
||||
readOnly: true
|
||||
- name: squid
|
||||
image: ubuntu/squid:latest
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- mkdir -p /tmp/squid && chmod 0777 /tmp/squid && /usr/sbin/squid -z -N -f /etc/squid/squid.conf && rm -f /run/squid.pid && exec /usr/sbin/squid -N -d 1
|
||||
ports:
|
||||
- name: proxy
|
||||
containerPort: 3128
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: false
|
||||
runAsUser: 13
|
||||
runAsGroup: 13
|
||||
volumeMounts:
|
||||
- name: squid
|
||||
mountPath: /etc/squid/squid.conf
|
||||
subPath: squid.conf
|
||||
readOnly: true
|
||||
- name: squid-run
|
||||
mountPath: /run
|
||||
|
||||
volumes:
|
||||
- name: client
|
||||
configMap:
|
||||
name: egress-check-client
|
||||
- name: squid
|
||||
configMap:
|
||||
name: egress-check-squid
|
||||
- name: squid-run
|
||||
emptyDir: {}
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: egress-check-goproxy
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: egress-check-goproxy-client
|
||||
namespace: egress-check-goproxy
|
||||
data:
|
||||
client.py: |
|
||||
import os
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
urls = [os.environ["CHECK_URL_1"], os.environ["CHECK_URL_2"]]
|
||||
timeout = float(os.environ.get("CHECK_TIMEOUT_SECONDS", "5"))
|
||||
interval = float(os.environ.get("CHECK_INTERVAL_SECONDS", "30"))
|
||||
while True:
|
||||
for url in urls:
|
||||
try:
|
||||
with urlopen(url, timeout=timeout) as response:
|
||||
print(url, response.status, flush=True)
|
||||
except Exception as error:
|
||||
print(url, error, flush=True)
|
||||
time.sleep(interval)
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: egress-check-goproxy
|
||||
namespace: egress-check-goproxy
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: egress-check-goproxy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: egress-check-goproxy
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
fsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
initContainers:
|
||||
- name: goproxy-download
|
||||
image: alpine:3.22@sha256:49340b346798875b5592e5d420241f2f9cfba5f47e62ebb1446a1b34734e898f
|
||||
command:
|
||||
- /bin/sh
|
||||
- -ec
|
||||
- |
|
||||
case "$(uname -m)" in
|
||||
x86_64)
|
||||
asset=proxy-linux-amd64.tar.gz
|
||||
checksum=00be86a17b16c4444fe86766e715c86a95d9e8e7bdf1b0b902ecd8bcf9cd8a54
|
||||
;;
|
||||
aarch64)
|
||||
asset=proxy-linux-arm64-v8.tar.gz
|
||||
checksum=dd230006c13caee98d66233a2b5764ad8d3d58e6ea081cec61344ada8f55d015
|
||||
;;
|
||||
armv7l)
|
||||
asset=proxy-linux-arm-v7.tar.gz
|
||||
checksum=78190e0c0ab4e979654a373d4677619e06ed962ba46bc0367398f92b010319c0
|
||||
;;
|
||||
*)
|
||||
echo "unsupported architecture: $(uname -m)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
wget -qO /tmp/proxy.tar.gz "https://github.com/snail007/goproxy/releases/download/v15.3/$asset"
|
||||
echo "$checksum /tmp/proxy.tar.gz" | sha256sum -c -
|
||||
tar -xzf /tmp/proxy.tar.gz -C /proxy
|
||||
chmod 0555 /proxy/proxy
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
volumeMounts:
|
||||
- name: goproxy
|
||||
mountPath: /proxy
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
containers:
|
||||
- name: client
|
||||
image: python:3.12-alpine
|
||||
command:
|
||||
- python
|
||||
- /app/client.py
|
||||
env:
|
||||
- name: CHECK_URL_1
|
||||
value: http://example.com/
|
||||
- name: CHECK_URL_2
|
||||
value: http://example.com/robots.txt
|
||||
- name: CHECK_INTERVAL_SECONDS
|
||||
value: "30"
|
||||
- name: CHECK_TIMEOUT_SECONDS
|
||||
value: "5"
|
||||
- name: http_proxy
|
||||
value: http://127.0.0.1:3128
|
||||
- name: https_proxy
|
||||
value: http://127.0.0.1:3128
|
||||
- name: HTTP_PROXY
|
||||
value: http://127.0.0.1:3128
|
||||
- name: HTTPS_PROXY
|
||||
value: http://127.0.0.1:3128
|
||||
- name: no_proxy
|
||||
value: ""
|
||||
- name: NO_PROXY
|
||||
value: ""
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
volumeMounts:
|
||||
- name: client
|
||||
mountPath: /app
|
||||
readOnly: true
|
||||
- name: goproxy
|
||||
image: alpine:3.22@sha256:49340b346798875b5592e5d420241f2f9cfba5f47e62ebb1446a1b34734e898f
|
||||
command:
|
||||
- /proxy/proxy
|
||||
- http
|
||||
- -t
|
||||
- tcp
|
||||
- -p
|
||||
- :3128
|
||||
ports:
|
||||
- name: proxy
|
||||
containerPort: 3128
|
||||
protocol: TCP
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
volumeMounts:
|
||||
- name: goproxy
|
||||
mountPath: /proxy
|
||||
readOnly: true
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
volumes:
|
||||
- name: goproxy
|
||||
emptyDir: {}
|
||||
- name: client
|
||||
configMap:
|
||||
name: egress-check-goproxy-client
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
+3
-4
@@ -30,7 +30,7 @@ metadata:
|
||||
app.kubernetes.io/name: k3s-connectivity
|
||||
spec:
|
||||
serviceName: k3s-connectivity
|
||||
replicas: 2
|
||||
replicas: 3
|
||||
podManagementPolicy: Parallel
|
||||
selector:
|
||||
matchLabels:
|
||||
@@ -80,13 +80,11 @@ spec:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
- name: REPLICAS
|
||||
value: "2"
|
||||
value: "3"
|
||||
- name: PROTOCOL
|
||||
value: http
|
||||
- name: PORT
|
||||
value: "8080"
|
||||
- name: PROBE_INTERVAL_SECONDS
|
||||
value: "30"
|
||||
- name: REQUEST_TIMEOUT_SECONDS
|
||||
value: "5"
|
||||
readinessProbe:
|
||||
@@ -113,6 +111,7 @@ spec:
|
||||
port: web
|
||||
scheme: HTTP
|
||||
periodSeconds: 2
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 15
|
||||
resources:
|
||||
requests:
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- kubernetes.yaml
|
||||
- kubernetes.yaml
|
||||
- egress-check.yaml
|
||||
- goproxy-egress-check.yaml
|
||||
|
||||
+42
-1
@@ -1,6 +1,11 @@
|
||||
import json
|
||||
import threading
|
||||
import unittest
|
||||
from http.server import ThreadingHTTPServer
|
||||
from urllib.request import urlopen
|
||||
|
||||
from app import Settings
|
||||
from app import (ConnectivityApplication, ConnectivityRequestHandler,
|
||||
PodIdentity, Settings)
|
||||
|
||||
|
||||
class SettingsTests(unittest.TestCase):
|
||||
@@ -27,5 +32,41 @@ class SettingsTests(unittest.TestCase):
|
||||
Settings.from_env({"PROTOCOL": "tcp"})
|
||||
|
||||
|
||||
class EndpointTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
settings = Settings.from_env({"REPLICAS": "1", "PORT": "8080"})
|
||||
application = ConnectivityApplication(
|
||||
settings,
|
||||
PodIdentity("k3s-connectivity-0", "test-node", "127.0.0.1"),
|
||||
)
|
||||
self.server = ThreadingHTTPServer(
|
||||
("127.0.0.1", 0), ConnectivityRequestHandler)
|
||||
self.server.daemon_threads = True
|
||||
setattr(self.server, "application", application)
|
||||
self.thread = threading.Thread(target=self.server.serve_forever)
|
||||
self.thread.start()
|
||||
self.base_url = f"http://127.0.0.1:{self.server.server_port}"
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.server.shutdown()
|
||||
self.server.server_close()
|
||||
self.thread.join()
|
||||
|
||||
def test_health_is_local_and_fast(self) -> None:
|
||||
with urlopen(f"{self.base_url}/healthz", timeout=1) as response:
|
||||
payload = json.loads(response.read())
|
||||
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(payload, {"status": "ok"})
|
||||
|
||||
def test_check_succeeds_without_peers(self) -> None:
|
||||
with urlopen(f"{self.base_url}/check", timeout=1) as response:
|
||||
payload = json.loads(response.read())
|
||||
|
||||
self.assertTrue(payload["all_reachable"])
|
||||
self.assertTrue(payload["all_cross_node"])
|
||||
self.assertEqual(payload["peers"], [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user