diff --git a/AGENTS.md b/AGENTS.md index 3c139dd..fe7f4ac 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,6 +66,13 @@ This file records repository decisions and the guidance that follows from them. - **Open options:** Narrow exports to exact client addresses, replace `no_root_squash` with `root_squash` after validating Hermes UID/GID behavior, or evaluate Kerberos-backed NFS with `sec=krb5`/`sec=krb5p` in a separate change. - **Status:** Draft; the current IP-restricted export model remains in use. +### DEC-012: Terminate public TLS in Traefik + +- **Context:** The public Kubernetes Ingress routes HTTP traffic through Traefik, while the KeenDNS/Keenetic HTTPS path does not reliably preserve the route to the requested host. Standard HTTPS URLs must work for the public domains without per-domain port suffixes. +- **Decision:** Forward WAN TCP `80` and `443` directly to `192.168.1.114` and terminate TLS in Traefik. Use Traefik's Let's Encrypt HTTP-01 resolver on the `web` entrypoint and the `websecure` entrypoint on `443`. Keenetic must not terminate TLS or rewrite SNI/Host. +- **Consequences:** Public port `80` is required for certificate issuance and renewal; ordinary HTTP requests are redirected to HTTPS. Traefik ACME storage must be persistent. Router changes remain manual; the operational procedure is documented in [`k3s_rencher/infrastructure/ingress/README.md`](k3s_rencher/infrastructure/ingress/README.md). +- **Status:** Proposed; apply only after the router forwarding, ACME email, persistent storage, and live Traefik chart configuration are verified. + ## Declined No declined decisions recorded. diff --git a/README.md b/README.md index 2832a27..a552610 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,8 @@ required follow-up actions. - [Gitea runner stack](./poc/gitea/stack-compose.yml) - [Hermes stack](./poc/hermes/README.md) - [Hermes Agent Kubernetes deployment](./k3s_rencher/apps/hermes/README.md) +- [Shared Traefik ingress and HTTPS operations](./k3s_rencher/infrastructure/ingress/README.md) +- [Traefik TLS/HTTP-01 decision](./docs/decisions/2026-11-21-traefik-tls-http01.md) - [Immich stack](./poc/immich/README.md) - [Lampa stack](./poc/lampa/README.md) - [NetBird deployment](./poc/netbird/README.md) diff --git a/docs/decisions/2026-11-21-traefik-tls-http01.md b/docs/decisions/2026-11-21-traefik-tls-http01.md new file mode 100644 index 0000000..4f7dec4 --- /dev/null +++ b/docs/decisions/2026-11-21-traefik-tls-http01.md @@ -0,0 +1,92 @@ +# Decision: TLS termination in Traefik on TCP 443 + +- Status: proposed +- Date: 2026-11-21 +- Scope: K3s edge ingress for public domains + +## Context + +The public HTTP routes are handled by the bundled Traefik instance on `xu4` +(`192.168.1.114`). The current public Ingress has HTTP rules for the Hermes, +Brencher, Immich, and Gitea hostnames. HTTPS is currently terminated by the +KeenDNS/Keenetic path and does not reliably reach the Traefik host rules. + +The required target is standard HTTPS with no port suffix: + +```text +https://immich.rudolf.keenetic.link/ +``` + +The same Traefik listener must support SNI/Host-based routing for the other +public hostnames. + +## Decision + +Use TCP forwarding from Keenetic to the Traefik node and terminate TLS in +Traefik: + +```text +WAN TCP 80 -> 192.168.1.114:80 +WAN TCP 443 -> 192.168.1.114:443 +``` + +Configure a Traefik `websecure` entrypoint on port `443` and use Traefik's +Let's Encrypt ACME resolver with the HTTP-01 challenge on the `web` entrypoint +(port `80`). Ordinary HTTP requests on `web` are permanently redirected to +`websecure`; Traefik's internal ACME challenge handling remains available for +HTTP-01 validation. Public Kubernetes Ingress resources will opt into TLS using +the same resolver. + +The router must not terminate TLS, use a KeenDNS HTTPS proxy, rewrite SNI, or +replace the HTTP `Host` header on the forwarded path. + +## Rationale + +- One standard HTTPS port supports all public domains. +- Traefik can select the certificate and route by SNI/Host. +- Certificate issuance and renewal are managed in one place. +- The design is independent of the Keenetic wildcard certificate. +- Swarm backends remain behind Traefik and are not exposed directly. + +## Consequences + +- WAN TCP `80` is a required dependency for every HTTP-01 validation and must + reach Traefik directly. +- ACME storage must be persistent across Traefik restarts and upgrades. +- A valid email address must be supplied to the ACME resolver configuration. +- Router-side changes are manual and are not represented by Kubernetes GitOps. +- The first rollout must be verified at DNS, TCP, ACME, TLS, Ingress, and + backend layers separately. + +## Rejected alternatives + +### TLS termination on Keenetic + +Rejected because it makes the router's TLS proxy the certificate owner and was +already observed to return the KeenDNS default `404` instead of reaching the +Traefik host route. + +### One public port per domain + +Rejected for the standard URL requirement. It would require URLs such as +`https://immich.rudolf.keenetic.link:8443/` and would not serve the default +port `443`. + +### DNS-01 + +Not selected because the requested operating model is HTTP-01. DNS-01 can be +considered later if direct public port `80` cannot be provided. + +## Acceptance criteria + +- `web` listens on TCP `80` and `websecure` listens on TCP `443`. +- Keenetic forwards TCP `80` and `443` to `192.168.1.114` without TLS + termination. +- The ACME storage is persistent and not committed with credentials. +- `curl -4I http://immich.rudolf.keenetic.link/` returns a permanent redirect to + the equivalent `https://` URL. +- HTTP-01 validation succeeds for every hostname requested in the Ingress. +- `curl --resolve :443: https:///` completes TLS with a + certificate valid for `` and reaches the expected backend. +- The ACME challenge path remains available on `web` and does not depend on the + KeenDNS web proxy. diff --git a/k3s_rencher/apps/hermes/ui.yaml b/k3s_rencher/apps/hermes/ui.yaml index 85711bf..278fdaf 100644 --- a/k3s_rencher/apps/hermes/ui.yaml +++ b/k3s_rencher/apps/hermes/ui.yaml @@ -4,8 +4,14 @@ metadata: name: hermes-ui labels: app.kubernetes.io/name: hermes + annotations: + traefik.ingress.kubernetes.io/router.tls: "true" + traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt spec: ingressClassName: traefik + tls: + - hosts: + - hermes.rudolf.keenetic.link rules: - host: hermes.rudolf.keenetic.link http: diff --git a/k3s_rencher/infrastructure/ingress/README.md b/k3s_rencher/infrastructure/ingress/README.md index 8e16c3b..f9f089a 100644 --- a/k3s_rencher/infrastructure/ingress/README.md +++ b/k3s_rencher/infrastructure/ingress/README.md @@ -1,7 +1,7 @@ # Shared public ingress The common Traefik entrypoint routes public domains to Kubernetes and Swarm -backends. The router forwards TCP port `80` to `xu4_lan` (`192.168.1.114`). +backends. | Domain | Backend | | --- | --- | @@ -13,3 +13,111 @@ backends. The router forwards TCP port `80` to `xu4_lan` (`192.168.1.114`). The Swarm backends are represented by selectorless Kubernetes Services and static Endpoints. The Swarm ports must remain reachable from the Traefik node at `192.168.1.96`. + +## HTTPS target configuration + +The target design is TCP forwarding to Traefik, with TLS termination in +Traefik: + +```text +WAN TCP 80 -> 192.168.1.114:80 +WAN TCP 443 -> 192.168.1.114:443 +``` + +Configure the Keenetic entries as raw TCP forwarding. Do not use the KeenDNS +HTTPS proxy, terminate TLS on the router, rewrite SNI, or replace the HTTP +`Host` header. + +Traefik must provide: + +- `web` on TCP `80` for HTTP and ACME HTTP-01; +- `websecure` on TCP `443` for TLS termination; +- a persistent ACME storage file, for example `/data/acme.json`; +- an ACME resolver with `httpChallenge.entryPoint: web`; +- a non-placeholder registration email supplied outside Git. + +The exact `HelmChartConfig` values must match the installed K3s Traefik chart +version. Inspect the live `HelmChart/traefik` values before applying a chart +override; do not assume that a values schema from another K3s release is +compatible. + +For each public Ingress, enable TLS with the same resolver. The resulting +Ingress shape is: + +```yaml +metadata: + annotations: + traefik.ingress.kubernetes.io/router.tls: "true" + traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt +spec: + tls: + - hosts: + - immich.rudolf.keenetic.link +``` + +With a Traefik ACME resolver, the certificate is stored in the configured +persistent ACME storage rather than committed to Git. If the installed chart +requires a Kubernetes Secret for a different TLS mode, provision that Secret +through the approved secret-management path. Do not commit `acme.json`, a +private key, an API token, or an unencrypted Kubernetes Secret. + +## Declarative traffic configuration + +The repository configuration is in `traefik-https-config.yaml`. It extends the +installed K3s `HelmChart/traefik` with: + +- `web` exposed on TCP `80`; +- `websecure` exposed on TCP `443`; +- permanent HTTP-to-HTTPS redirection from `web` to `websecure`; +- the `letsencrypt` ACME resolver using HTTP-01 through `web` (Traefik's + ACME challenge handling remains available on the `web` entrypoint); +- persistent `/data/acme.json` storage through a `1Gi` PVC. + +The resolver uses the operator-supplied registration email from the +configuration. The email is not a credential; the ACME account state and +certificate private keys remain in persistent `/data/acme.json` storage and +must not be committed to Git. The TLS annotations and host lists are present +in both the shared public Ingress and Hermes-specific Ingress. + +## Rollout procedure + +1. Verify that every hostname resolves to the router's current public address. +2. Configure Keenetic WAN TCP `80` and `443` forwarding to + `192.168.1.114:80` and `192.168.1.114:443`. +3. Confirm from an external network that TCP `80` reaches Traefik. The + KeenDNS default `404` is not proof of correct forwarding. +4. Configure the Traefik `websecure` entrypoint, persistent ACME storage, and + HTTP-01 resolver through the repository's GitOps path. +5. Add TLS annotations and host entries to the public Ingress resources. +6. Apply through Flux only after rendering and reviewing the complete diff. +7. Check the Traefik logs and ACME resource/state for the challenge result. +8. Test HTTP-01 reachability: + + ```sh + curl -4v http://immich.rudolf.keenetic.link/.well-known/acme-challenge/test + ``` + + This request must reach Traefik, not the KeenDNS web proxy. + +9. Test TLS and SNI directly against the WAN address: + + ```sh + curl -4vk \ + --resolve immich.rudolf.keenetic.link:443: \ + https://immich.rudolf.keenetic.link/ + ``` + +10. Verify the certificate SAN, the Traefik route, the HTTP status, and the + Immich backend separately. Repeat the TLS/SNI test for every public + hostname. + +## Failure boundaries + +- DNS resolves incorrectly: fix DNS before changing Kubernetes. +- TCP `80` or `443` is refused: fix Keenetic forwarding or WAN reachability. +- KeenDNS certificate/default `404` appears: traffic still uses the router + proxy instead of TCP forwarding. +- ACME challenge fails: inspect port `80`, Host preservation, and Traefik + resolver/storage before changing backend routes. +- TLS succeeds but the route is `404`/`502`: inspect the Ingress, Service, + Endpoints, and Traefik access log separately. diff --git a/k3s_rencher/infrastructure/ingress/kustomization.yaml b/k3s_rencher/infrastructure/ingress/kustomization.yaml index 309c3e6..9628179 100644 --- a/k3s_rencher/infrastructure/ingress/kustomization.yaml +++ b/k3s_rencher/infrastructure/ingress/kustomization.yaml @@ -2,4 +2,5 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - swarm-backends.yaml + - traefik-https-config.yaml - public-ingress.yaml diff --git a/k3s_rencher/infrastructure/ingress/public-ingress.yaml b/k3s_rencher/infrastructure/ingress/public-ingress.yaml index ddf2c6f..a9ef04e 100644 --- a/k3s_rencher/infrastructure/ingress/public-ingress.yaml +++ b/k3s_rencher/infrastructure/ingress/public-ingress.yaml @@ -5,8 +5,16 @@ metadata: namespace: hermes labels: app.kubernetes.io/name: public-ingress + annotations: + traefik.ingress.kubernetes.io/router.tls: "true" + traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt spec: ingressClassName: traefik + tls: + - hosts: + - brencher.rudolf.keenetic.link + - immich.rudolf.keenetic.link + - gitea.rudolf.keenetic.link rules: - host: brencher.rudolf.keenetic.link http: diff --git a/k3s_rencher/infrastructure/ingress/traefik-https-config.yaml b/k3s_rencher/infrastructure/ingress/traefik-https-config.yaml new file mode 100644 index 0000000..d46b0c9 --- /dev/null +++ b/k3s_rencher/infrastructure/ingress/traefik-https-config.yaml @@ -0,0 +1,39 @@ +apiVersion: helm.cattle.io/v1 +kind: HelmChartConfig +metadata: + name: traefik + namespace: kube-system +spec: + valuesContent: |- + nodeSelector: + kubernetes.io/hostname: xu4 + deployment: + replicas: 1 + ports: + web: + port: 8000 + exposedPort: 80 + expose: + default: true + protocol: TCP + websecure: + port: 8443 + exposedPort: 443 + expose: + default: true + protocol: TCP + additionalArguments: + - "--entrypoints.web.address=:8000/tcp" + - "--entrypoints.websecure.address=:8443/tcp" + - "--entrypoints.web.http.redirections.entrypoint.to=websecure" + - "--entrypoints.web.http.redirections.entrypoint.scheme=https" + - "--entrypoints.web.http.redirections.entrypoint.permanent=true" + - "--certificatesresolvers.letsencrypt.acme.email=rudolfss13@gmail.com" + - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json" + - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web" + persistence: + enabled: true + name: data + path: /data + accessMode: ReadWriteOnce + size: 1Gi