forked from org1/infra
Compare commits
24
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0f9c5fa105 | ||
|
|
24198a56e4 | ||
|
|
e084ce4ecf | ||
|
|
696aacfabe | ||
|
|
c0e042f0a0 | ||
|
|
95edba82bf | ||
|
|
5e314e17d0 | ||
|
|
5e45811200 | ||
|
|
6a7ad7e235 | ||
|
|
2c84082e22 | ||
|
|
7a029d5c85 | ||
|
|
410a0be8e3 | ||
|
|
4e1dfc905f | ||
|
|
5ba8ae21e0 | ||
|
|
5b5ede0b65 | ||
|
|
b0c5413bf7 | ||
|
|
9a0551192a | ||
|
|
e1a7faf1fb | ||
|
|
f4c54f8010 | ||
|
|
5cb4642d67 | ||
|
|
2889600de1 | ||
|
|
7789e95247 | ||
|
|
ca63dd6069 | ||
|
|
fc6f668a86 |
@@ -1,5 +0,0 @@
|
||||
# Project Instructions
|
||||
|
||||
## Docker Swarm Stack Files
|
||||
|
||||
All YAML files in this project are Docker Swarm stack files. Keep them compatible with Docker Swarm and deploy them with `docker stack deploy -c <file> <stack-name>`, not `docker compose up`.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Repository Decisions
|
||||
|
||||
This file records repository decisions and the guidance that follows from them. Accepted decisions are in force. Draft decisions are proposals, not policy. Declined decisions are retained for context and are not in force.
|
||||
|
||||
## Accepted
|
||||
|
||||
### DEC-001: Make the Kubernetes cluster the primary infrastructure
|
||||
|
||||
- **Decision:** New services should be deployed in the Kubernetes cluster, and services running on other instances should be migrated there.
|
||||
[kustomization.yaml](kustomization.yaml) is entry point for the Kubernetes cluster configuration.
|
||||
|
||||
### DEC-002: Document manual operations in README.md
|
||||
|
||||
- **Decision:** The repository's `README.md` should document useful manual operations and link to all other manual instructions in the repository.
|
||||
|
||||
### DEC-003: Retire Ansible
|
||||
|
||||
- **Decision:** Ansible is legacy and should be removed strategically.
|
||||
|
||||
### DEC-004: Treat Docker Swarm files as legacy
|
||||
|
||||
- **Decision:** Docker Swarm files are legacy. New services should not use Docker Swarm and should adopt Kubernetes or other modern orchestration solutions.
|
||||
|
||||
### DEC-005: Isolate Kubernetes namespaces by default
|
||||
|
||||
- **Decision:** Each namespace should be isolated by default for ingress and egress. Egress should allow only explicitly whitelisted connectivity.
|
||||
|
||||
### DEC-006: Use GitOps for infrastructure changes
|
||||
|
||||
- **Decision:** Use GitOps as much as possible and keep the repository as the source for infrastructure. Automate changes through CI/CD. Make infrastructure changes through pull requests and have them reviewed before application.
|
||||
|
||||
### DEC-007: Preserve Docker Swarm deployment for legacy stack files
|
||||
|
||||
- **Decision:** Legacy Docker Swarm stack files should remain compatible with Docker Swarm and be deployed with `docker stack deploy -c <file> <stack-name>`, not `docker compose up`.
|
||||
|
||||
### DEC-008: Apply configuration through Git pushes only
|
||||
|
||||
- **Decision:** Configuration should be applied only through Git pushes. Do not apply configuration directly to infrastructure, even when access is available. An agent may apply configs directly tom infrastructure only when explicitly requested.
|
||||
|
||||
### DEC-009: Use separate NFS state for Kubernetes Hermes
|
||||
|
||||
- **Context:** Kubernetes Hermes must not share mutable state with the legacy Swarm Hermes instance. K3s `local-path` storage is node-local and does not provide the desired storage layout.
|
||||
- **Decision:** Kubernetes Hermes uses a static NFS-backed `hermes-data-nfs` claim from `192.168.1.96:/root/volumes/hermes/k3s`, mounted at `/opt/data`. The volume uses NFSv3 and `Retain` reclaim policy. The Swarm directories `/root/volumes/hermes/gateway` and `/root/volumes/hermes/agent` remain separate.
|
||||
- **Consequences:** The NFS export must allow the current Kubernetes node addresses `100.70.193.97`, `100.105.201.15`, and `100.96.92.102`, in addition to the existing Swarm LAN allowance `192.168.1.0/24`. Kubernetes and Swarm Hermes must not use the same mutable state directory concurrently.
|
||||
|
||||
### DEC-010: Manage web application domains through Keenetic Hero 4G
|
||||
|
||||
- **Context:** Remote access to web applications in the network is managed through the Keenetic Hero 4G router's web application access settings.
|
||||
- **Decision:** The router-managed web application mappings are:
|
||||
|
||||
| Domain | Remote access | Host | IP address | Protocol | TCP port |
|
||||
| --- | --- | --- | --- | --- | ---: |
|
||||
| `brencher.rudolf.keenetic.link` | Authorized | `odroid` | `192.168.1.96` | HTTP | `5001` |
|
||||
| `immich.rudolf.keenetic.link` | Public | `odroid` | `192.168.1.96` | HTTP | `2283` |
|
||||
| `gitea.rudolf.keenetic.link` | No access | `odroid` | `192.168.1.96` | HTTP | `3002` |
|
||||
| `hermes.rudolf.keenetic.link` | Public | `xu4_lan` | `192.168.1.114` | HTTP | `80` |
|
||||
|
||||
- **Consequences:** These entries describe HTTP/HTTPS web application publishing through the router. They do not imply that arbitrary TCP ports, including Docker API ports `2375` or `2376`, are published. Raw TCP access requires a separate router port-forward or another explicitly configured network path.
|
||||
|
||||
## Draft
|
||||
|
||||
### DEC-011: Add authenticated NFS access for Hermes
|
||||
|
||||
- **Context:** The current NFS access model relies on export client allowlists and Unix filesystem permissions. It does not provide NFS username/password authentication.
|
||||
- **Decision:** Defer NFS authentication. Do not add Kerberos or store NFS credentials in Kubernetes as part of the current Hermes storage migration.
|
||||
- **Open options:** Narrow exports to exact client addresses, replace `no_root_squash` with `root_squash` after validating Hermes UID/GID behavior, or evaluate Kerberos-backed NFS with `sec=krb5`/`sec=krb5p` in a separate change.
|
||||
- **Status:** Draft; the current IP-restricted export model remains in use.
|
||||
|
||||
### DEC-012: Terminate public TLS in Traefik
|
||||
|
||||
- **Context:** The public Kubernetes Ingress routes HTTP traffic through Traefik, while the KeenDNS/Keenetic HTTPS path does not reliably preserve the route to the requested host. Standard HTTPS URLs must work for the public domains without per-domain port suffixes.
|
||||
- **Decision:** Forward WAN TCP `80` and `443` directly to `192.168.1.114` and terminate TLS in Traefik. Use Traefik's Let's Encrypt HTTP-01 resolver on the `web` entrypoint and the `websecure` entrypoint on `443`. Keenetic must not terminate TLS or rewrite SNI/Host.
|
||||
- **Consequences:** Public port `80` is required for certificate issuance and renewal; ordinary HTTP requests are redirected to HTTPS. Traefik ACME storage must be persistent. Router changes remain manual; the operational procedure is documented in [`k3s_rencher/infrastructure/ingress/README.md`](k3s_rencher/infrastructure/ingress/README.md).
|
||||
- **Status:** Proposed; apply only after the router forwarding, ACME email, persistent storage, and live Traefik chart configuration are verified.
|
||||
|
||||
## Declined
|
||||
|
||||
No declined decisions recorded.
|
||||
|
||||
## Adding a decision
|
||||
|
||||
Add each decision under its current status. Use a unique `DEC-NNN` identifier and include the context, decision, and consequences. Drafts are not instructions; move a decision to Accepted when approved, or to Declined if it is not adopted.
|
||||
|
||||
### DEC-NNN: Short title
|
||||
|
||||
- **Context:** Why a decision is needed.
|
||||
- **Decision:** What is decided.
|
||||
- **Consequences:** Important effects or tradeoffs.
|
||||
+161
@@ -0,0 +1,161 @@
|
||||
# Open findings
|
||||
|
||||
This file tracks findings from the repository compliance review against the
|
||||
accepted decisions in [AGENTS.md](./AGENTS.md). The root
|
||||
[README.md](./README.md) now documents the repository and manual operations;
|
||||
the findings below are the remaining unresolved items.
|
||||
|
||||
## DEC-001 — Kubernetes is not yet the primary infrastructure
|
||||
|
||||
**Status:** Open
|
||||
|
||||
The root [kustomization.yaml](./kustomization.yaml) currently reconciles Flux,
|
||||
DNS, the agent gateway, and network-policy scaffolding, but no production
|
||||
application workloads. Production services are still documented as Docker
|
||||
Swarm deployments in [PROD.md](./PROD.md), and service definitions remain
|
||||
under [poc/](./poc/), [nextcloud/](./nextcloud/), and
|
||||
[hermes_squid/](./hermes_squid/).
|
||||
|
||||
Required follow-up:
|
||||
|
||||
- Define target Kubernetes namespaces and ownership for production services.
|
||||
- Add application overlays under `k3s_rencher/`.
|
||||
- Migrate services to Kubernetes one at a time.
|
||||
- Add migrated workloads to the root Kustomize entry point.
|
||||
- Replace production Swarm instructions after each successful migration.
|
||||
|
||||
## DEC-003 — Ansible has not been retired
|
||||
|
||||
**Status:** Open
|
||||
|
||||
The repository still contains active-looking playbooks, inventory, variables,
|
||||
Galaxy dependencies, and host-management roles:
|
||||
|
||||
- [all.yaml](./all.yaml)
|
||||
- [gogs.yaml](./gogs.yaml)
|
||||
- [orientdb.yaml](./orientdb.yaml)
|
||||
- [syncthing.yaml](./syncthing.yaml)
|
||||
- [hosts](./hosts)
|
||||
- [group_vars/all](./group_vars/all)
|
||||
- [requirements.yaml](./requirements.yaml)
|
||||
- [roles/](./roles/)
|
||||
|
||||
Required follow-up:
|
||||
|
||||
- Inventory all Ansible-managed resources.
|
||||
- Classify each resource as migrated, required for host bootstrap, obsolete,
|
||||
or requiring a Kubernetes replacement.
|
||||
- Retain only unavoidable host bootstrap.
|
||||
- Deprecate and then remove obsolete playbooks, roles, inventory, and
|
||||
dependencies after replacement procedures are documented.
|
||||
|
||||
## DEC-004 — Swarm remains the production deployment path
|
||||
|
||||
**Status:** Open
|
||||
|
||||
Legacy Swarm files may remain during migration, but [PROD.md](./PROD.md) still
|
||||
instructs operators to initialize Swarm and deploy production services with
|
||||
`docker stack deploy`. Service documentation also presents Swarm or Docker
|
||||
Compose as the primary deployment method.
|
||||
|
||||
Relevant legacy material includes:
|
||||
|
||||
- [docker-compose-nginx.yaml](./docker-compose-nginx.yaml)
|
||||
- [docker-compose-registry.yaml](./docker-compose-registry.yaml)
|
||||
- [nextcloud/stack-compose.yaml](./nextcloud/stack-compose.yaml)
|
||||
- [hermes_squid/stack-compose.yml](./hermes_squid/stack-compose.yml)
|
||||
- [poc/immich/stack-compose.yml](./poc/immich/stack-compose.yml)
|
||||
- [poc/gitea/stack-compose.yml](./poc/gitea/stack-compose.yml)
|
||||
- [poc/vscode/stack-code-server.yml](./poc/vscode/stack-code-server.yml)
|
||||
|
||||
Required follow-up:
|
||||
|
||||
- Mark remaining Swarm files and instructions as legacy and identify their
|
||||
migration status.
|
||||
- Do not add new Swarm services.
|
||||
- Migrate production services to Kubernetes and replace their operational
|
||||
instructions.
|
||||
- Keep legacy stack files compatible with `docker stack deploy` until removed.
|
||||
|
||||
## DEC-005 — Namespace isolation is incomplete
|
||||
|
||||
**Status:** Partially implemented
|
||||
|
||||
The default policies in
|
||||
[default-deny.yaml](./k3s_rencher/infrastructure/network-policies/default-deny.yaml)
|
||||
and the DNS exception in
|
||||
[allow-dns.yaml](./k3s_rencher/infrastructure/network-policies/allow-dns.yaml)
|
||||
are currently applied through the `agent-gateway` overlay only.
|
||||
|
||||
Required follow-up:
|
||||
|
||||
- Create a dedicated Kustomize overlay for every workload namespace.
|
||||
- Apply default-deny ingress and egress to every workload namespace.
|
||||
- Add explicit ingress, pod-to-pod, and external egress policies per service.
|
||||
- Decide and document the policy baseline for `flux-system` and other system
|
||||
namespaces.
|
||||
- Verify that the externally sourced connectivity tester has equivalent
|
||||
namespace and network-policy controls.
|
||||
- Review all future uses of `hostNetwork`, host ports, privileged containers,
|
||||
and node-originated traffic.
|
||||
|
||||
## DEC-006 — GitOps changes are not sufficiently validated or gated
|
||||
|
||||
**Status:** Partially implemented
|
||||
|
||||
Flux reconciliation is configured through
|
||||
[flux-system/gitrepository.yaml](./k3s_rencher/flux-system/gitrepository.yaml)
|
||||
and [flux-system/sync.yaml](./k3s_rencher/flux-system/sync.yaml), but the
|
||||
repository has no visible pull-request validation workflow for Kubernetes
|
||||
changes. The existing
|
||||
[release workflow](./.gitea/workflows/release.yml) only builds and publishes
|
||||
the connectivity tester image.
|
||||
|
||||
Required follow-up:
|
||||
|
||||
- Add pull-request checks for YAML, Kustomize rendering, Kubernetes schemas,
|
||||
network policies, secrets, and image references.
|
||||
- Protect the Flux-tracked branch and require review plus passing checks.
|
||||
- Document the one-time bootstrap boundary versus normal Flux-managed changes.
|
||||
- Add rollback and reconciliation verification procedures.
|
||||
- Review whether public Git sources are an intentional trust decision.
|
||||
- Prefer immutable image tags or digests over `latest` for production.
|
||||
|
||||
## Secret and credential hygiene
|
||||
|
||||
**Status:** Open
|
||||
|
||||
Potential credentials and unsafe examples remain in tracked files, including
|
||||
[roles/metricbeat/defaults/main.yml](./roles/metricbeat/defaults/main.yml),
|
||||
[roles/orientdb/defaults/main.yml](./roles/orientdb/defaults/main.yml),
|
||||
[scratches/tomcat/tomcat-users.xml](./scratches/tomcat/tomcat-users.xml),
|
||||
and [hermes_squid/README.md](./hermes_squid/README.md).
|
||||
|
||||
Required follow-up:
|
||||
|
||||
- Determine whether each value was ever active.
|
||||
- Rotate any potentially active credential.
|
||||
- Remove credentials from tracked files and examples.
|
||||
- Use SOPS, External Secrets, or another approved secret manager for
|
||||
Kubernetes workloads.
|
||||
- Keep only clearly fake placeholders in documentation.
|
||||
|
||||
## Repository structure and lifecycle
|
||||
|
||||
**Status:** Open
|
||||
|
||||
The repository is still structured primarily as an Ansible project even though
|
||||
the accepted direction is Kubernetes and GitOps. Kubernetes resources,
|
||||
experimental deployments, legacy stacks, and historical scratch files are
|
||||
mixed together.
|
||||
|
||||
Required follow-up:
|
||||
|
||||
- Establish clear ownership boundaries for cluster bootstrap,
|
||||
infrastructure, applications, legacy deployments, and historical material.
|
||||
- Consider separating `cluster-bootstrap/`, `infrastructure/`, `apps/`, and
|
||||
`legacy/`.
|
||||
- Archive or remove obsolete material only after its replacement and rollback
|
||||
procedure are documented.
|
||||
- Keep the root [README.md](./README.md) and this file updated as migrations
|
||||
complete.
|
||||
@@ -0,0 +1,46 @@
|
||||
# Infrastructure repository
|
||||
|
||||
See [FINDINGS.md](./FINDINGS.md) for unresolved compliance findings and
|
||||
required follow-up actions.
|
||||
|
||||
|
||||
## Manual operations and service documentation
|
||||
|
||||
### Cluster and production operations
|
||||
|
||||
- [K3s installation and health operations](./k3s_rencher/README.md)
|
||||
- [GitOps and Flux bootstrap](./k3s_rencher/README-gitops.md)
|
||||
- [Legacy production setup](./PROD.md)
|
||||
- [Legacy Docker Compose operations](./README_docker_compose.md)
|
||||
|
||||
### Service documentation
|
||||
|
||||
- [Nextcloud operations](./nextcloud/README.md)
|
||||
- [Hermes Squid operations](./hermes_squid/README.md)
|
||||
- [Anywherelan deployment](./poc/anywherelan/README.md)
|
||||
- [Beszel stack](./poc/beszel/stack-compose.yml)
|
||||
- [Gitea runner stack](./poc/gitea/stack-compose.yml)
|
||||
- [Hermes stack](./poc/hermes/README.md)
|
||||
- [Hermes Agent Kubernetes deployment](./k3s_rencher/apps/hermes/README.md)
|
||||
- [Shared Traefik ingress and HTTPS operations](./k3s_rencher/infrastructure/ingress/README.md)
|
||||
- [Traefik TLS/HTTP-01 decision](./docs/decisions/2026-11-21-traefik-tls-http01.md)
|
||||
- [Immich stack](./poc/immich/README.md)
|
||||
- [Lampa stack](./poc/lampa/README.md)
|
||||
- [NetBird deployment](./poc/netbird/README.md)
|
||||
- [VS Code server stack](./poc/vscode/README.md)
|
||||
|
||||
The service-specific Docker documentation remains valid only for services
|
||||
that have not yet been migrated. Update the migration status above and the
|
||||
service documentation when a service moves to Kubernetes.
|
||||
|
||||
## Repository layout
|
||||
|
||||
- [k3s_rencher/](./k3s_rencher/) — Kubernetes and Flux resources.
|
||||
- [connectivity_tester/](./connectivity_tester/) — connectivity tester image
|
||||
and tests.
|
||||
- [poc/](./poc/) — legacy and experimental service deployments.
|
||||
- [nextcloud/](./nextcloud/) and [hermes_squid/](./hermes_squid/) — legacy
|
||||
service deployments.
|
||||
- [roles/](./roles/) — legacy Ansible roles retained during migration.
|
||||
- [scratches/](./scratches/) — historical experiments; not production
|
||||
configuration.
|
||||
@@ -0,0 +1,92 @@
|
||||
# Decision: TLS termination in Traefik on TCP 443
|
||||
|
||||
- Status: proposed
|
||||
- Date: 2026-11-21
|
||||
- Scope: K3s edge ingress for public domains
|
||||
|
||||
## Context
|
||||
|
||||
The public HTTP routes are handled by the bundled Traefik instance on `xu4`
|
||||
(`192.168.1.114`). The current public Ingress has HTTP rules for the Hermes,
|
||||
Brencher, Immich, and Gitea hostnames. HTTPS is currently terminated by the
|
||||
KeenDNS/Keenetic path and does not reliably reach the Traefik host rules.
|
||||
|
||||
The required target is standard HTTPS with no port suffix:
|
||||
|
||||
```text
|
||||
https://immich.rudolf.keenetic.link/
|
||||
```
|
||||
|
||||
The same Traefik listener must support SNI/Host-based routing for the other
|
||||
public hostnames.
|
||||
|
||||
## Decision
|
||||
|
||||
Use TCP forwarding from Keenetic to the Traefik node and terminate TLS in
|
||||
Traefik:
|
||||
|
||||
```text
|
||||
WAN TCP 80 -> 192.168.1.114:80
|
||||
WAN TCP 443 -> 192.168.1.114:443
|
||||
```
|
||||
|
||||
Configure a Traefik `websecure` entrypoint on port `443` and use Traefik's
|
||||
Let's Encrypt ACME resolver with the HTTP-01 challenge on the `web` entrypoint
|
||||
(port `80`). Ordinary HTTP routes use a separate permanent `redirectScheme`
|
||||
middleware on `web`; TLS routes are pinned to `websecure`. This keeps the
|
||||
internal ACME challenge router separate from backend routes. Public Kubernetes
|
||||
Ingress resources will opt into TLS using the same resolver.
|
||||
|
||||
The router must not terminate TLS, use a KeenDNS HTTPS proxy, rewrite SNI, or
|
||||
replace the HTTP `Host` header on the forwarded path.
|
||||
|
||||
## Rationale
|
||||
|
||||
- One standard HTTPS port supports all public domains.
|
||||
- Traefik can select the certificate and route by SNI/Host.
|
||||
- Certificate issuance and renewal are managed in one place.
|
||||
- The design is independent of the Keenetic wildcard certificate.
|
||||
- Swarm backends remain behind Traefik and are not exposed directly.
|
||||
|
||||
## Consequences
|
||||
|
||||
- WAN TCP `80` is a required dependency for every HTTP-01 validation and must
|
||||
reach Traefik directly.
|
||||
- ACME storage must be persistent across Traefik restarts and upgrades.
|
||||
- A valid email address must be supplied to the ACME resolver configuration.
|
||||
- Router-side changes are manual and are not represented by Kubernetes GitOps.
|
||||
- The first rollout must be verified at DNS, TCP, ACME, TLS, Ingress, and
|
||||
backend layers separately.
|
||||
|
||||
## Rejected alternatives
|
||||
|
||||
### TLS termination on Keenetic
|
||||
|
||||
Rejected because it makes the router's TLS proxy the certificate owner and was
|
||||
already observed to return the KeenDNS default `404` instead of reaching the
|
||||
Traefik host route.
|
||||
|
||||
### One public port per domain
|
||||
|
||||
Rejected for the standard URL requirement. It would require URLs such as
|
||||
`https://immich.rudolf.keenetic.link:8443/` and would not serve the default
|
||||
port `443`.
|
||||
|
||||
### DNS-01
|
||||
|
||||
Not selected because the requested operating model is HTTP-01. DNS-01 can be
|
||||
considered later if direct public port `80` cannot be provided.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `web` listens on TCP `80` and `websecure` listens on TCP `443`.
|
||||
- Keenetic forwards TCP `80` and `443` to `192.168.1.114` without TLS
|
||||
termination.
|
||||
- The ACME storage is persistent and not committed with credentials.
|
||||
- `curl -4I http://immich.rudolf.keenetic.link/` returns a permanent redirect to
|
||||
the equivalent `https://` URL.
|
||||
- HTTP-01 validation succeeds for every hostname requested in the Ingress.
|
||||
- `curl --resolve <host>:443:<wan-ip> https://<host>/` completes TLS with a
|
||||
certificate valid for `<host>` and reaches the expected backend.
|
||||
- The ACME challenge path remains available on `web` and does not depend on the
|
||||
KeenDNS web proxy.
|
||||
@@ -40,17 +40,17 @@ flux get kustomizations -n flux-system
|
||||
|
||||
The pinned v2.6.4 release is installed as a separate bootstrap step because
|
||||
the Flux CRDs must exist before the `GitRepository` and `Kustomization`
|
||||
objects can be applied. The root Kustomization then creates the source and reconciliation objects for
|
||||
the K3s `main` branch at the repository root and the connectivity tester
|
||||
`main` branch under `./k3s`.
|
||||
objects can be applied. The root Kustomization then creates the source and
|
||||
reconciliation objects for this repository's `main` branch at the repository
|
||||
root and the connectivity tester `main` branch under `./k3s`.
|
||||
|
||||
## Local bootstrap resources
|
||||
|
||||
The cluster configuration is `k3s_rencher/`. Render it before applying:
|
||||
The cluster configuration is the repository root. Render it before applying:
|
||||
|
||||
```sh
|
||||
kubectl kustomize k3s_rencher
|
||||
kubectl apply -k k3s_rencher
|
||||
kubectl kustomize .
|
||||
kubectl apply -k .
|
||||
```
|
||||
|
||||
The overlay currently installs the read-only agent account and its network
|
||||
|
||||
@@ -24,6 +24,24 @@ Run on `rudolf-T`:
|
||||
|
||||
```sh
|
||||
export K3S_TOKEN='<TODO>'
|
||||
sudo env K3S_TOKEN="$K3S_TOKEN" sh -c '
|
||||
set -eu
|
||||
|
||||
install -d -m 0755 /etc/rancher/k3s
|
||||
install -d -m 0755 /etc/systemd/system/k3s-agent.service.d
|
||||
|
||||
printf "%s\\n" \
|
||||
"nameserver 192.168.1.1" \
|
||||
"options timeout:2 attempts:2" \
|
||||
> /etc/rancher/k3s-resolv.conf
|
||||
|
||||
printf "%s\\n" \
|
||||
"[Service]" \
|
||||
"BindReadOnlyPaths=/etc/rancher/k3s-resolv.conf:/etc/resolv.conf" \
|
||||
> /etc/systemd/system/k3s-agent.service.d/10-registry-dns.conf
|
||||
|
||||
systemctl daemon-reload
|
||||
|
||||
curl -sfL https://get.k3s.io | \
|
||||
K3S_URL="https://100.70.193.97:6443" \
|
||||
K3S_TOKEN="$K3S_TOKEN" \
|
||||
@@ -31,8 +49,11 @@ curl -sfL https://get.k3s.io | \
|
||||
--node-ip="100.105.201.15" \
|
||||
--node-external-ip="100.105.201.15" \
|
||||
--flannel-iface="tailscale0"
|
||||
'
|
||||
```
|
||||
|
||||
The resolver override is mounted only into the `k3s-agent` service namespace. It does not replace the Ubuntu host resolver for other services. The embedded containerd inherits it for image pulls.
|
||||
|
||||
### Join `xu4`
|
||||
|
||||
Run on `xu4`:
|
||||
@@ -70,3 +91,13 @@ kubectl get pods -A -o wide
|
||||
|
||||
# Recent warnings/errors across namespaces
|
||||
kubectl get events -A --sort-by=.lastTimestamp | tail -n 50
|
||||
|
||||
# Request a fresh Git fetch. The source update triggers its dependent Kustomization.
|
||||
kubectl annotate -n flux-system --overwrite gitrepository/k3s reconcile.fluxcd.io/requestedAt="$(date +%s)"
|
||||
|
||||
# Compare the latest fetched commit with the last successful application.
|
||||
kubectl get kustomization k3s -n flux-system -o jsonpath='applied: {.status.lastAppliedRevision}{"\nattempted: "}{.status.lastAttemptedRevision}{"\n"}'
|
||||
|
||||
# Show reconciliation conditions and recent errors/events.
|
||||
kubectl describe kustomization k3s -n flux-system
|
||||
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
# Hermes Swarm to K3s Migration
|
||||
|
||||
## Placement and storage
|
||||
|
||||
Placement matches the Swarm constraints:
|
||||
|
||||
- Hermes and the UI are scheduled on `rudolf-t`.
|
||||
|
||||
The Kubernetes deployment uses a static NFS-backed claim:
|
||||
|
||||
- `hermes-data-nfs` -> `/root/volumes/hermes/k3s` -> `/opt/data`
|
||||
|
||||
The directory is separate from the Swarm Hermes paths:
|
||||
|
||||
- `/root/volumes/hermes/gateway`
|
||||
- `/root/volumes/hermes/agent`
|
||||
|
||||
The old `hermes-data` local-path PVC remains declared for rollback and is not
|
||||
mounted by the Deployment after the NFS switch.
|
||||
|
||||
Verify the claims and volume before migration:
|
||||
|
||||
```sh
|
||||
kubectl get pvc -n hermes
|
||||
kubectl get pv
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
Create the NFS directory on `192.168.1.96` before rollout:
|
||||
|
||||
```sh
|
||||
sudo mkdir -p /root/volumes/hermes/k3s
|
||||
```
|
||||
|
||||
The NFS claim is intentionally separate from the Swarm state. Copy data only
|
||||
when the Kubernetes instance should inherit a selected subset of the Swarm
|
||||
state; do not run both instances against the same mutable directory.
|
||||
|
||||
Do not run both stacks against the same Telegram bot token or the same
|
||||
mutable data directory at the same time.
|
||||
|
||||
## Copy Docker volumes into PVCs
|
||||
|
||||
Run this before applying the Hermes Deployment. First identify the real
|
||||
Docker volume names; Swarm usually prefixes them with the stack name:
|
||||
|
||||
```sh
|
||||
docker volume ls
|
||||
docker volume inspect <stack>_hermes_gateway_data
|
||||
```
|
||||
|
||||
Stop the Swarm stack and make sure no K3s Hermes workloads are using the PVCs:
|
||||
|
||||
```sh
|
||||
docker stack services <stack>
|
||||
docker stack rm <stack>
|
||||
kubectl -n hermes get pods
|
||||
```
|
||||
|
||||
Create a temporary pod that mounts the claim. The node affinity ensures the
|
||||
local-path provisioner creates it on `rudolf-t`:
|
||||
|
||||
```sh
|
||||
cat <<'EOF' | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: hermes-volume-migration
|
||||
namespace: hermes
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: kubernetes.io/hostname
|
||||
operator: In
|
||||
values:
|
||||
- rudolf-t
|
||||
containers:
|
||||
- name: migration
|
||||
image: alpine:3.20
|
||||
command: ["sleep", "3600"]
|
||||
volumeMounts:
|
||||
- name: hermes-data
|
||||
mountPath: /target/hermes
|
||||
volumes:
|
||||
- name: hermes-data
|
||||
persistentVolumeClaim:
|
||||
claimName: hermes-data
|
||||
EOF
|
||||
|
||||
kubectl -n hermes wait --for=condition=Ready pod/hermes-volume-migration \
|
||||
--timeout=180s
|
||||
```
|
||||
|
||||
From the Docker host, copy the volume as a tar stream. Replace the volume
|
||||
name with the name reported by `docker volume ls`:
|
||||
|
||||
```sh
|
||||
docker run --rm \
|
||||
-v '<stack>_hermes_gateway_data:/source:ro' \
|
||||
alpine:3.20 tar -C /source -cf - . |
|
||||
kubectl -n hermes exec -i hermes-volume-migration -- \
|
||||
tar -C /target/hermes -xf -
|
||||
|
||||
```
|
||||
|
||||
Verify the copied data before removing the temporary pod:
|
||||
|
||||
```sh
|
||||
kubectl -n hermes exec hermes-volume-migration -- du -sh /target/hermes
|
||||
kubectl -n hermes delete pod hermes-volume-migration
|
||||
```
|
||||
|
||||
Finally deploy Hermes and verify the rollout:
|
||||
|
||||
```sh
|
||||
kubectl apply -k .
|
||||
kubectl -n hermes rollout status deployment/hermes
|
||||
kubectl -n hermes get pods,pvc
|
||||
```
|
||||
|
||||
If Docker runs on a different host than `kubectl`, run the `docker run ...`
|
||||
part on that host and pipe its output over SSH to the machine running
|
||||
`kubectl`, for example:
|
||||
|
||||
```sh
|
||||
ssh <docker-host> "docker run --rm -v '<stack>_hermes_gateway_data:/source:ro' alpine:3.20 tar -C /source -cf -" |
|
||||
kubectl -n hermes exec -i hermes-volume-migration -- tar -C /target/hermes -xf -
|
||||
```
|
||||
@@ -0,0 +1,52 @@
|
||||
# Hermes Agent
|
||||
|
||||
This deployment runs the Hermes Agent gateway and its Nginx dashboard reverse
|
||||
proxy in Kubernetes with persistent state.
|
||||
|
||||
Hermes is a single gateway process; Telegram, the dashboard, scheduled jobs,
|
||||
and persisted sessions share the same pod.
|
||||
|
||||
## Credentials
|
||||
|
||||
Create `hermes-credentials` in the `hermes` namespace with the credentials
|
||||
appropriate for the selected model provider:
|
||||
|
||||
```sh
|
||||
kubectl -n hermes create secret generic hermes-credentials \
|
||||
--from-literal=HERMES_DASHBOARD_BASIC_AUTH_USERNAME='<dashboard user>' \
|
||||
--from-literal=HERMES_DASHBOARD_BASIC_AUTH_PASSWORD='<dashboard password>'
|
||||
```
|
||||
|
||||
Do not commit this Secret; supply it through the cluster's secret-management
|
||||
workflow.
|
||||
|
||||
Hermes egress allows public IPv4 destinations while blocking:
|
||||
|
||||
- `192.168.0.0/24`
|
||||
- `192.168.1.0/24`
|
||||
- `172.0.0.0/8`
|
||||
- `100.0.0.0/8`
|
||||
|
||||
The policy is defined in [network-policy.yaml](./network-policy.yaml).
|
||||
|
||||
## Migration
|
||||
|
||||
See [Swarm to K3s migration](./MIGRATION.md) for placement, volume, and
|
||||
cutover instructions.
|
||||
|
||||
## Deploy
|
||||
|
||||
Render and inspect the manifests, then let Flux reconcile them:
|
||||
|
||||
```sh
|
||||
kubectl kustomize k3s_rencher/apps/hermes
|
||||
kubectl apply -k k3s_rencher/apps/hermes
|
||||
kubectl -n hermes rollout status deployment/hermes
|
||||
```
|
||||
|
||||
The dashboard is exposed through the cluster Traefik Ingress at
|
||||
`hermes.rudolf.keenetic.link`. The Ingress routes to the `hermes` Service on
|
||||
port `9119`; Traefik is allowed to reach that port by the namespace NetworkPolicy.
|
||||
The router must keep the public web application entry pointed at
|
||||
`xu4_lan` (`192.168.1.114`) on HTTP port `80`. Do not point the router at the
|
||||
Kubernetes Service port `9119` or at a Tailscale node address.
|
||||
@@ -0,0 +1,77 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: hermes
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: hermes
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: kubernetes.io/hostname
|
||||
operator: In
|
||||
values:
|
||||
- rudolf-t
|
||||
securityContext:
|
||||
runAsNonRoot: false
|
||||
runAsUser: 0
|
||||
runAsGroup: 0
|
||||
containers:
|
||||
- name: hermes
|
||||
image: nousresearch/hermes-agent:v2026.8.3
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- gateway
|
||||
- run
|
||||
env:
|
||||
- name: HERMES_DASHBOARD
|
||||
value: "1"
|
||||
- name: HERMES_DASHBOARD_HOST
|
||||
value: 0.0.0.0
|
||||
- name: HERMES_YOLO_MODE
|
||||
value: "1"
|
||||
- name: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hermes-credentials
|
||||
key: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
|
||||
- name: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hermes-credentials
|
||||
key: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
|
||||
ports:
|
||||
- name: dashboard
|
||||
containerPort: 9119
|
||||
protocol: TCP
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /opt/data
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
add:
|
||||
- CHOWN
|
||||
- DAC_OVERRIDE
|
||||
- SETGID
|
||||
- SETUID
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: hermes-data-nfs
|
||||
@@ -0,0 +1,97 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: brencher-state-proxy-config
|
||||
data:
|
||||
default.conf: |
|
||||
server {
|
||||
listen 8080;
|
||||
server_name _;
|
||||
location = /healthz {
|
||||
access_log off;
|
||||
return 200 "ok\n";
|
||||
add_header Content-Type text/plain;
|
||||
}
|
||||
location = /state {
|
||||
proxy_pass https://brencher.rudolf.keenetic.link/state;
|
||||
proxy_http_version 1.1;
|
||||
proxy_ssl_server_name on;
|
||||
proxy_ssl_name brencher.rudolf.keenetic.link;
|
||||
proxy_set_header Host brencher.rudolf.keenetic.link;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_connect_timeout 5s;
|
||||
proxy_read_timeout 30s;
|
||||
proxy_send_timeout 10s;
|
||||
proxy_buffering off;
|
||||
add_header Cache-Control "no-store" always;
|
||||
}
|
||||
location = /state2 {
|
||||
proxy_pass http://192.168.1.96:5002/state;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_connect_timeout 5s;
|
||||
proxy_read_timeout 30s;
|
||||
proxy_send_timeout 10s;
|
||||
proxy_buffering off;
|
||||
add_header Cache-Control "no-store" always;
|
||||
}
|
||||
location / {
|
||||
access_log off;
|
||||
return 403 "forbidden: only /state and /state2 are allowed through this proxy\n";
|
||||
add_header Content-Type text/plain;
|
||||
}
|
||||
}
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brencher-state-proxy
|
||||
labels:
|
||||
app.kubernetes.io/name: brencher-state-proxy
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: brencher-state-proxy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: brencher-state-proxy
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
protocol: TCP
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/nginx/conf.d/default.conf
|
||||
subPath: default.conf
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: brencher-state-proxy-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brencher-state-proxy
|
||||
labels:
|
||||
app.kubernetes.io/name: brencher-state-proxy
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: brencher-state-proxy
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
@@ -0,0 +1,24 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: docker-socket-proxy
|
||||
labels:
|
||||
app.kubernetes.io/name: docker-socket-proxy
|
||||
spec:
|
||||
ports:
|
||||
- name: docker
|
||||
port: 2375
|
||||
targetPort: 2375
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Endpoints
|
||||
metadata:
|
||||
name: docker-socket-proxy
|
||||
subsets:
|
||||
- addresses:
|
||||
- ip: 192.168.1.96
|
||||
ports:
|
||||
- name: docker
|
||||
port: 2375
|
||||
protocol: TCP
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: hermes-gateway
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: hermes-data-nfs
|
||||
@@ -0,0 +1,14 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: hermes-gateway
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes-gateway
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: hermes-gateway
|
||||
ports:
|
||||
- name: dashboard
|
||||
port: 9119
|
||||
targetPort: dashboard
|
||||
protocol: TCP
|
||||
@@ -0,0 +1,91 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: hermes-gateway
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes-gateway
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: hermes-gateway
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes-gateway
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: kubernetes.io/hostname
|
||||
operator: In
|
||||
values:
|
||||
- rudolf-t
|
||||
securityContext:
|
||||
runAsNonRoot: false
|
||||
runAsUser: 0
|
||||
runAsGroup: 0
|
||||
containers:
|
||||
- name: hermes-gateway
|
||||
image: nousresearch/hermes-agent:v2026.8.3
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- gateway
|
||||
- run
|
||||
env:
|
||||
- name: HERMES_DASHBOARD
|
||||
value: "1"
|
||||
- name: HERMES_DASHBOARD_HOST
|
||||
value: 0.0.0.0
|
||||
- name: HERMES_YOLO_MODE
|
||||
value: "1"
|
||||
- name: HERMES_AGENT_URL
|
||||
value: http://hermes-agent:8000
|
||||
- name: HTTP_PROXY
|
||||
value: http://squid:3128
|
||||
- name: HTTPS_PROXY
|
||||
value: http://squid:3128
|
||||
- name: http_proxy
|
||||
value: http://squid:3128
|
||||
- name: https_proxy
|
||||
value: http://squid:3128
|
||||
- name: NO_PROXY
|
||||
value: localhost,127.0.0.1,.local,squid,hermes-gateway,hermes-agent,hermes-ui,brencher-state-proxy,docker-socket-proxy,10.0.0.0/8
|
||||
- name: no_proxy
|
||||
value: localhost,127.0.0.1,.local,squid,hermes-gateway,hermes-agent,hermes-ui,brencher-state-proxy,docker-socket-proxy,10.0.0.0/8
|
||||
- name: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hermes-credentials
|
||||
key: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
|
||||
- name: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hermes-credentials
|
||||
key: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
|
||||
ports:
|
||||
- name: dashboard
|
||||
containerPort: 9119
|
||||
protocol: TCP
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /opt/data
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
add:
|
||||
- CHOWN
|
||||
- DAC_OVERRIDE
|
||||
- SETGID
|
||||
- SETUID
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: hermes-data
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- gateway.yaml
|
||||
- gateway-service.yaml
|
||||
- ui.yaml
|
||||
- squid.yaml
|
||||
- docker-socket-proxy.yaml
|
||||
- brencher-state-proxy.yaml
|
||||
- network-policy.yaml
|
||||
patches:
|
||||
- path: gateway-pvc-patch.yaml
|
||||
@@ -0,0 +1,124 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: hermes-egress
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: hermes-gateway
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.1.1/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 100.70.193.97/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 192.168.0.0/24
|
||||
- 192.168.1.0/24
|
||||
- 172.0.0.0/8
|
||||
- 100.0.0.0/8
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.1.96/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 2375
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 100.105.201.15/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9875
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: hermes-ingress
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: hermes-gateway
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/instance: traefik-kube-system
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9119
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: hermes-ui
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9119
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: brencher-state-proxy-egress
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: brencher-state-proxy
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 192.168.0.0/24
|
||||
- 192.168.1.0/24
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.1.96/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5002
|
||||
@@ -0,0 +1,80 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: squid-config
|
||||
data:
|
||||
squid.conf: |
|
||||
acl localnet src 10.0.0.0/8
|
||||
acl localnet src 172.16.0.0/12
|
||||
acl localnet src 192.168.0.0/16
|
||||
acl allowed_domains dstdomain .minimax.io .minimax.chat .minimax.com .githubusercontent.com deb.debian.org gitea.rudolf.keenetic.link
|
||||
acl telegram_domains dstdomain .telegram.org .t.me
|
||||
acl stt_pkg_domains dstdomain .pypi.org .files.pythonhosted.org .huggingface.co .github.com .models.dev .registry.npmjs.org .cdn-lfs.huggingface.co .hf.co .xethub.hf.co .cas-server.xethub.hf.co
|
||||
acl copilot_domains dstdomain api.github.com copilot-proxy.githubusercontent.com .githubcopilot.com copilot-telemetry.githubusercontent.com default.exp-tas.com marketplace.visualstudio.com vscode.blob.core.windows.net .githubassets.com .vscode-unpkg.net
|
||||
acl telegram_ips dst 91.108.4.0/22 91.108.8.0/22 91.108.12.0/22 91.108.16.0/22 91.108.20.0/22 91.108.56.0/22 149.154.160.0/20 185.76.151.0/24
|
||||
acl CONNECT method CONNECT
|
||||
acl SSL_ports port 443
|
||||
acl Safe_ports port 80 443
|
||||
http_access deny !Safe_ports
|
||||
http_access deny CONNECT !SSL_ports
|
||||
http_access allow localnet allowed_domains
|
||||
http_access allow localnet telegram_domains
|
||||
http_access allow localnet stt_pkg_domains
|
||||
http_access allow localnet copilot_domains
|
||||
http_access allow localnet telegram_ips
|
||||
http_access deny all
|
||||
http_port 3128
|
||||
access_log stdio:/var/log/squid/access.log
|
||||
cache deny all
|
||||
cache_mem 0 MB
|
||||
cache_dir null /tmp
|
||||
coredump_dir /var/spool/squid
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: squid
|
||||
labels:
|
||||
app.kubernetes.io/name: squid
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: squid
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: squid
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
containers:
|
||||
- name: squid
|
||||
image: ubuntu/squid:latest
|
||||
ports:
|
||||
- name: proxy
|
||||
containerPort: 3128
|
||||
protocol: TCP
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/squid/squid.conf
|
||||
subPath: squid.conf
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: squid-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: squid
|
||||
labels:
|
||||
app.kubernetes.io/name: squid
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: squid
|
||||
ports:
|
||||
- name: proxy
|
||||
port: 3128
|
||||
targetPort: proxy
|
||||
protocol: TCP
|
||||
@@ -0,0 +1,19 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: hermes-ui
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: hermes.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: hermes
|
||||
port:
|
||||
name: dashboard
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: hermes-generic1
|
||||
resources:
|
||||
- ../generic-base
|
||||
- namespace.yaml
|
||||
- nfs-pv.yaml
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: hermes-generic1
|
||||
@@ -0,0 +1,29 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: hermes-generic1-data-nfs
|
||||
spec:
|
||||
capacity:
|
||||
storage: 10Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: hermes-generic1-nfs
|
||||
mountOptions:
|
||||
- nfsvers=3
|
||||
nfs:
|
||||
server: 192.168.1.96
|
||||
path: /root/volumes/hermes/generic1
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: hermes-data-nfs
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: hermes-generic1-nfs
|
||||
volumeName: hermes-generic1-data-nfs
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: hermes-generic2
|
||||
resources:
|
||||
- ../generic-base
|
||||
- namespace.yaml
|
||||
- nfs-pv.yaml
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: hermes-generic2
|
||||
@@ -0,0 +1,29 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: hermes-generic2-data-nfs
|
||||
spec:
|
||||
capacity:
|
||||
storage: 10Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: hermes-generic2-nfs
|
||||
mountOptions:
|
||||
- nfsvers=3
|
||||
nfs:
|
||||
server: 192.168.1.96
|
||||
path: /root/volumes/hermes/generic2
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: hermes-data-nfs
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: hermes-generic2-nfs
|
||||
volumeName: hermes-generic2-data-nfs
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: hermes
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- pvc.yaml
|
||||
- nfs-pv.yaml
|
||||
- nfs-pvc.yaml
|
||||
- ui.yaml
|
||||
- deployment.yaml
|
||||
- service.yaml
|
||||
- network-policy.yaml
|
||||
@@ -0,0 +1,4 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: hermes
|
||||
@@ -0,0 +1,88 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: hermes-egress
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: hermes
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
k8s-app: kube-dns
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.1.1/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 100.70.193.97/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 192.168.0.0/24
|
||||
- 192.168.1.0/24
|
||||
- 172.0.0.0/8
|
||||
- 100.0.0.0/8
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 192.168.1.96/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 2375
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 100.105.201.15/32
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9875
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: hermes-ingress
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: hermes
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: hermes
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9119
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/instance: traefik-kube-system
|
||||
app.kubernetes.io/name: traefik
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9119
|
||||
@@ -0,0 +1,16 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: hermes-data-nfs
|
||||
spec:
|
||||
capacity:
|
||||
storage: 10Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: hermes-nfs
|
||||
mountOptions:
|
||||
- nfsvers=3
|
||||
nfs:
|
||||
server: 192.168.1.96
|
||||
path: /root/volumes/hermes/k3s
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: hermes-data-nfs
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: hermes-nfs
|
||||
volumeName: hermes-data-nfs
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
@@ -0,0 +1,10 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: hermes-data
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
@@ -0,0 +1,14 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: hermes
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: hermes
|
||||
ports:
|
||||
- name: dashboard
|
||||
port: 9119
|
||||
targetPort: dashboard
|
||||
protocol: TCP
|
||||
@@ -0,0 +1,26 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: hermes-ui
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- hermes.rudolf.keenetic.link
|
||||
rules:
|
||||
- host: hermes.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: hermes
|
||||
port:
|
||||
name: dashboard
|
||||
@@ -7,4 +7,4 @@ spec:
|
||||
interval: 1m
|
||||
ref:
|
||||
branch: main
|
||||
url: https://gitea.rudolf.keenetic.link/org1/k3s.git
|
||||
url: https://gitea.rudolf.keenetic.link/org1/infra.git
|
||||
|
||||
@@ -5,14 +5,108 @@ metadata:
|
||||
namespace: agent-gateway
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: agent-readonly-all-except-secrets
|
||||
rules:
|
||||
- apiGroups:
|
||||
- ""
|
||||
resources:
|
||||
- bindings
|
||||
- componentstatuses
|
||||
- configmaps
|
||||
- endpoints
|
||||
- events
|
||||
- limitranges
|
||||
- namespaces
|
||||
- namespaces/status
|
||||
- nodes
|
||||
- nodes/status
|
||||
- persistentvolumeclaims
|
||||
- persistentvolumeclaims/status
|
||||
- persistentvolumes
|
||||
- persistentvolumes/status
|
||||
- pods
|
||||
- pods/ephemeralcontainers
|
||||
- pods/log
|
||||
- pods/status
|
||||
- podtemplates
|
||||
- replicationcontrollers
|
||||
- replicationcontrollers/status
|
||||
- resourcequotas
|
||||
- resourcequotas/status
|
||||
- serviceaccounts
|
||||
- services
|
||||
- services/status
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
- apiGroups:
|
||||
- apiregistration.k8s.io
|
||||
- apps
|
||||
- admissionregistration.k8s.io
|
||||
- apiextensions.k8s.io
|
||||
- authentication.k8s.io
|
||||
- authorization.k8s.io
|
||||
- autoscaling
|
||||
- batch
|
||||
- certificates.k8s.io
|
||||
- coordination.k8s.io
|
||||
- discovery.k8s.io
|
||||
- events.k8s.io
|
||||
- flowcontrol.apiserver.k8s.io
|
||||
- gateway.networking.k8s.io
|
||||
- helm.cattle.io
|
||||
- helm.toolkit.fluxcd.io
|
||||
- hub.traefik.io
|
||||
- image.toolkit.fluxcd.io
|
||||
- k3s.cattle.io
|
||||
- kustomize.toolkit.fluxcd.io
|
||||
- metrics.k8s.io
|
||||
- networking.k8s.io
|
||||
- node.k8s.io
|
||||
- notification.toolkit.fluxcd.io
|
||||
- policy
|
||||
- rbac.authorization.k8s.io
|
||||
- resource.k8s.io
|
||||
- scheduling.k8s.io
|
||||
- source.toolkit.fluxcd.io
|
||||
- storage.k8s.io
|
||||
- traefik.io
|
||||
resources:
|
||||
- "*"
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
- nonResourceURLs:
|
||||
- /api
|
||||
- /api/*
|
||||
- /apis
|
||||
- /apis/*
|
||||
- /healthz
|
||||
- /healthz/*
|
||||
- /livez
|
||||
- /livez/*
|
||||
- /openapi
|
||||
- /openapi/*
|
||||
- /readyz
|
||||
- /readyz/*
|
||||
- /version
|
||||
- /version/*
|
||||
verbs:
|
||||
- get
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: agent-readonly-view
|
||||
name: agent-readonly-all-except-secrets
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: view
|
||||
name: agent-readonly-all-except-secrets
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: agent-readonly
|
||||
namespace: agent-gateway
|
||||
namespace: agent-gateway
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
# Shared public ingress
|
||||
|
||||
The common Traefik entrypoint routes public domains to Kubernetes and Swarm
|
||||
backends.
|
||||
|
||||
| Domain | Backend |
|
||||
| --- | --- |
|
||||
| `hermes.rudolf.keenetic.link` | Kubernetes Service `hermes:9119` |
|
||||
| `brencher.rudolf.keenetic.link` | Swarm `192.168.1.96:5001` |
|
||||
| `immich.rudolf.keenetic.link` | Swarm `192.168.1.96:2283` |
|
||||
| `gitea.rudolf.keenetic.link` | Swarm `192.168.1.96:3002` |
|
||||
|
||||
The Swarm backends are represented by selectorless Kubernetes Services and
|
||||
static Endpoints. The Swarm ports must remain reachable from the Traefik node
|
||||
at `192.168.1.96`.
|
||||
|
||||
## HTTPS target configuration
|
||||
|
||||
The target design is TCP forwarding to Traefik, with TLS termination in
|
||||
Traefik:
|
||||
|
||||
```text
|
||||
WAN TCP 80 -> 192.168.1.114:80
|
||||
WAN TCP 443 -> 192.168.1.114:443
|
||||
```
|
||||
|
||||
Configure the Keenetic entries as raw TCP forwarding. Do not use the KeenDNS
|
||||
HTTPS proxy, terminate TLS on the router, rewrite SNI, or replace the HTTP
|
||||
`Host` header.
|
||||
|
||||
Traefik must provide:
|
||||
|
||||
- `web` on TCP `80` for HTTP and ACME HTTP-01;
|
||||
- `websecure` on TCP `443` for TLS termination;
|
||||
- a persistent ACME storage file, for example `/data/acme.json`;
|
||||
- an ACME resolver with `httpChallenge.entryPoint: web`;
|
||||
- a non-placeholder registration email supplied outside Git.
|
||||
|
||||
The exact `HelmChartConfig` values must match the installed K3s Traefik chart
|
||||
version. Inspect the live `HelmChart/traefik` values before applying a chart
|
||||
override; do not assume that a values schema from another K3s release is
|
||||
compatible.
|
||||
|
||||
For each public Ingress, pin the TLS route to `websecure`. The separate
|
||||
`http-redirect.yaml` resource attaches a `redirectScheme` middleware to
|
||||
`web`-only routes. Keeping the HTTP redirect separate prevents the internal
|
||||
ACME challenge router from being sent to a backend.
|
||||
|
||||
The resulting TLS Ingress shape is:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
|
||||
spec:
|
||||
tls:
|
||||
- hosts:
|
||||
- immich.rudolf.keenetic.link
|
||||
```
|
||||
|
||||
With a Traefik ACME resolver, the certificate is stored in the configured
|
||||
persistent ACME storage rather than committed to Git. If the installed chart
|
||||
requires a Kubernetes Secret for a different TLS mode, provision that Secret
|
||||
through the approved secret-management path. Do not commit `acme.json`, a
|
||||
private key, an API token, or an unencrypted Kubernetes Secret.
|
||||
|
||||
## Declarative traffic configuration
|
||||
|
||||
The repository configuration is in `traefik-https-config.yaml`. It extends the
|
||||
installed K3s `HelmChart/traefik` with:
|
||||
|
||||
- `web` exposed on TCP `80` for the HTTP redirect and ACME HTTP-01;
|
||||
- `websecure` exposed on TCP `443` for TLS termination;
|
||||
- a permanent HTTP-to-HTTPS `redirectScheme` middleware on web-only routes;
|
||||
- the `letsencrypt` ACME resolver using HTTP-01 through `web` (Traefik's
|
||||
ACME challenge handling remains available on the `web` entrypoint);
|
||||
- persistent `/data/acme.json` storage through a `1Gi` PVC.
|
||||
|
||||
The resolver uses the operator-supplied registration email from the
|
||||
configuration. The email is not a credential; the ACME account state and
|
||||
certificate private keys remain in persistent `/data/acme.json` storage and
|
||||
must not be committed to Git. The TLS annotations and host lists are present
|
||||
in both the shared public Ingress and Hermes-specific Ingress.
|
||||
|
||||
## Rollout procedure
|
||||
|
||||
1. Verify that every hostname resolves to the router's current public address.
|
||||
2. Configure Keenetic WAN TCP `80` and `443` forwarding to
|
||||
`192.168.1.114:80` and `192.168.1.114:443`.
|
||||
3. Confirm from an external network that TCP `80` reaches Traefik. The
|
||||
KeenDNS default `404` is not proof of correct forwarding.
|
||||
4. Configure the Traefik `websecure` entrypoint, persistent ACME storage, and
|
||||
HTTP-01 resolver through the repository's GitOps path.
|
||||
5. Add TLS annotations and host entries to the public Ingress resources.
|
||||
6. Apply through Flux only after rendering and reviewing the complete diff.
|
||||
7. Check the Traefik logs and ACME resource/state for the challenge result.
|
||||
8. Test HTTP-01 reachability:
|
||||
|
||||
```sh
|
||||
curl -4v http://immich.rudolf.keenetic.link/.well-known/acme-challenge/test
|
||||
```
|
||||
|
||||
This request must reach Traefik, not the KeenDNS web proxy.
|
||||
|
||||
9. Test TLS and SNI directly against the WAN address:
|
||||
|
||||
```sh
|
||||
curl -4vk \
|
||||
--resolve immich.rudolf.keenetic.link:443:<wan-ip> \
|
||||
https://immich.rudolf.keenetic.link/
|
||||
```
|
||||
|
||||
10. Verify the certificate SAN, the Traefik route, the HTTP status, and the
|
||||
Immich backend separately. Repeat the TLS/SNI test for every public
|
||||
hostname.
|
||||
|
||||
## Failure boundaries
|
||||
|
||||
- DNS resolves incorrectly: fix DNS before changing Kubernetes.
|
||||
- TCP `80` or `443` is refused: fix Keenetic forwarding or WAN reachability.
|
||||
- KeenDNS certificate/default `404` appears: traffic still uses the router
|
||||
proxy instead of TCP forwarding.
|
||||
- ACME challenge fails: inspect port `80`, Host preservation, and Traefik
|
||||
resolver/storage before changing backend routes.
|
||||
- TLS succeeds but the route is `404`/`502`: inspect the Ingress, Service,
|
||||
Endpoints, and Traefik access log separately.
|
||||
@@ -0,0 +1,63 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: Middleware
|
||||
metadata:
|
||||
name: http-to-https
|
||||
namespace: hermes
|
||||
spec:
|
||||
redirectScheme:
|
||||
scheme: https
|
||||
permanent: true
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: http-redirect
|
||||
namespace: hermes
|
||||
labels:
|
||||
app.kubernetes.io/name: http-redirect
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: web
|
||||
traefik.ingress.kubernetes.io/router.middlewares: hermes-http-to-https@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: hermes.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: hermes
|
||||
port:
|
||||
name: dashboard
|
||||
- host: brencher.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: brencher-swarm
|
||||
port:
|
||||
name: http
|
||||
- host: immich.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: immich-swarm
|
||||
port:
|
||||
name: http
|
||||
- host: gitea.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: gitea-swarm
|
||||
port:
|
||||
name: http
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- swarm-backends.yaml
|
||||
- traefik-https-config.yaml
|
||||
- http-redirect.yaml
|
||||
- public-ingress.yaml
|
||||
@@ -0,0 +1,49 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: public-ingress
|
||||
namespace: hermes
|
||||
labels:
|
||||
app.kubernetes.io/name: public-ingress
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- brencher.rudolf.keenetic.link
|
||||
- immich.rudolf.keenetic.link
|
||||
- gitea.rudolf.keenetic.link
|
||||
rules:
|
||||
- host: brencher.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: brencher-swarm
|
||||
port:
|
||||
name: http
|
||||
- host: immich.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: immich-swarm
|
||||
port:
|
||||
name: http
|
||||
- host: gitea.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: gitea-swarm
|
||||
port:
|
||||
name: http
|
||||
@@ -0,0 +1,80 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
namespace: hermes
|
||||
name: brencher-swarm
|
||||
labels:
|
||||
app.kubernetes.io/name: brencher-swarm
|
||||
spec:
|
||||
ports:
|
||||
- name: http
|
||||
port: 5001
|
||||
targetPort: 5001
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Endpoints
|
||||
metadata:
|
||||
namespace: hermes
|
||||
name: brencher-swarm
|
||||
subsets:
|
||||
- addresses:
|
||||
- ip: 192.168.1.96
|
||||
ports:
|
||||
- name: http
|
||||
port: 5001
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
namespace: hermes
|
||||
name: immich-swarm
|
||||
labels:
|
||||
app.kubernetes.io/name: immich-swarm
|
||||
spec:
|
||||
ports:
|
||||
- name: http
|
||||
port: 2283
|
||||
targetPort: 2283
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Endpoints
|
||||
metadata:
|
||||
namespace: hermes
|
||||
name: immich-swarm
|
||||
subsets:
|
||||
- addresses:
|
||||
- ip: 192.168.1.96
|
||||
ports:
|
||||
- name: http
|
||||
port: 2283
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
namespace: hermes
|
||||
name: gitea-swarm
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea-swarm
|
||||
spec:
|
||||
ports:
|
||||
- name: http
|
||||
port: 3002
|
||||
targetPort: 3002
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Endpoints
|
||||
metadata:
|
||||
namespace: hermes
|
||||
name: gitea-swarm
|
||||
subsets:
|
||||
- addresses:
|
||||
- ip: 192.168.1.96
|
||||
ports:
|
||||
- name: http
|
||||
port: 3002
|
||||
protocol: TCP
|
||||
@@ -0,0 +1,36 @@
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChartConfig
|
||||
metadata:
|
||||
name: traefik
|
||||
namespace: kube-system
|
||||
spec:
|
||||
valuesContent: |-
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: xu4
|
||||
deployment:
|
||||
replicas: 1
|
||||
ports:
|
||||
web:
|
||||
port: 8000
|
||||
exposedPort: 80
|
||||
expose:
|
||||
default: true
|
||||
protocol: TCP
|
||||
websecure:
|
||||
port: 8443
|
||||
exposedPort: 443
|
||||
expose:
|
||||
default: true
|
||||
protocol: TCP
|
||||
additionalArguments:
|
||||
- "--entrypoints.web.address=:8000/tcp"
|
||||
- "--entrypoints.websecure.address=:8443/tcp"
|
||||
- "--certificatesresolvers.letsencrypt.acme.email=rudolfss13@gmail.com"
|
||||
- "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
|
||||
- "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
|
||||
persistence:
|
||||
enabled: true
|
||||
name: data
|
||||
path: /data
|
||||
accessMode: ReadWriteOnce
|
||||
size: 1Gi
|
||||
@@ -4,3 +4,7 @@ resources:
|
||||
- k3s_rencher/flux-system
|
||||
- k3s_rencher/infrastructure/dns
|
||||
- k3s_rencher/infrastructure/agent-gateway
|
||||
- k3s_rencher/infrastructure/ingress
|
||||
- k3s_rencher/apps/hermes
|
||||
- k3s_rencher/apps/hermes/generic1
|
||||
- k3s_rencher/apps/hermes/generic2
|
||||
|
||||
Reference in New Issue
Block a user