forked from org1/infra
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0f9c5fa105 | ||
|
|
24198a56e4 |
@@ -66,6 +66,13 @@ This file records repository decisions and the guidance that follows from them.
|
||||
- **Open options:** Narrow exports to exact client addresses, replace `no_root_squash` with `root_squash` after validating Hermes UID/GID behavior, or evaluate Kerberos-backed NFS with `sec=krb5`/`sec=krb5p` in a separate change.
|
||||
- **Status:** Draft; the current IP-restricted export model remains in use.
|
||||
|
||||
### DEC-012: Terminate public TLS in Traefik
|
||||
|
||||
- **Context:** The public Kubernetes Ingress routes HTTP traffic through Traefik, while the KeenDNS/Keenetic HTTPS path does not reliably preserve the route to the requested host. Standard HTTPS URLs must work for the public domains without per-domain port suffixes.
|
||||
- **Decision:** Forward WAN TCP `80` and `443` directly to `192.168.1.114` and terminate TLS in Traefik. Use Traefik's Let's Encrypt HTTP-01 resolver on the `web` entrypoint and the `websecure` entrypoint on `443`. Keenetic must not terminate TLS or rewrite SNI/Host.
|
||||
- **Consequences:** Public port `80` is required for certificate issuance and renewal; ordinary HTTP requests are redirected to HTTPS. Traefik ACME storage must be persistent. Router changes remain manual; the operational procedure is documented in [`k3s_rencher/infrastructure/ingress/README.md`](k3s_rencher/infrastructure/ingress/README.md).
|
||||
- **Status:** Proposed; apply only after the router forwarding, ACME email, persistent storage, and live Traefik chart configuration are verified.
|
||||
|
||||
## Declined
|
||||
|
||||
No declined decisions recorded.
|
||||
|
||||
@@ -22,6 +22,8 @@ required follow-up actions.
|
||||
- [Gitea runner stack](./poc/gitea/stack-compose.yml)
|
||||
- [Hermes stack](./poc/hermes/README.md)
|
||||
- [Hermes Agent Kubernetes deployment](./k3s_rencher/apps/hermes/README.md)
|
||||
- [Shared Traefik ingress and HTTPS operations](./k3s_rencher/infrastructure/ingress/README.md)
|
||||
- [Traefik TLS/HTTP-01 decision](./docs/decisions/2026-11-21-traefik-tls-http01.md)
|
||||
- [Immich stack](./poc/immich/README.md)
|
||||
- [Lampa stack](./poc/lampa/README.md)
|
||||
- [NetBird deployment](./poc/netbird/README.md)
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# Decision: TLS termination in Traefik on TCP 443
|
||||
|
||||
- Status: proposed
|
||||
- Date: 2026-11-21
|
||||
- Scope: K3s edge ingress for public domains
|
||||
|
||||
## Context
|
||||
|
||||
The public HTTP routes are handled by the bundled Traefik instance on `xu4`
|
||||
(`192.168.1.114`). The current public Ingress has HTTP rules for the Hermes,
|
||||
Brencher, Immich, and Gitea hostnames. HTTPS is currently terminated by the
|
||||
KeenDNS/Keenetic path and does not reliably reach the Traefik host rules.
|
||||
|
||||
The required target is standard HTTPS with no port suffix:
|
||||
|
||||
```text
|
||||
https://immich.rudolf.keenetic.link/
|
||||
```
|
||||
|
||||
The same Traefik listener must support SNI/Host-based routing for the other
|
||||
public hostnames.
|
||||
|
||||
## Decision
|
||||
|
||||
Use TCP forwarding from Keenetic to the Traefik node and terminate TLS in
|
||||
Traefik:
|
||||
|
||||
```text
|
||||
WAN TCP 80 -> 192.168.1.114:80
|
||||
WAN TCP 443 -> 192.168.1.114:443
|
||||
```
|
||||
|
||||
Configure a Traefik `websecure` entrypoint on port `443` and use Traefik's
|
||||
Let's Encrypt ACME resolver with the HTTP-01 challenge on the `web` entrypoint
|
||||
(port `80`). Ordinary HTTP routes use a separate permanent `redirectScheme`
|
||||
middleware on `web`; TLS routes are pinned to `websecure`. This keeps the
|
||||
internal ACME challenge router separate from backend routes. Public Kubernetes
|
||||
Ingress resources will opt into TLS using the same resolver.
|
||||
|
||||
The router must not terminate TLS, use a KeenDNS HTTPS proxy, rewrite SNI, or
|
||||
replace the HTTP `Host` header on the forwarded path.
|
||||
|
||||
## Rationale
|
||||
|
||||
- One standard HTTPS port supports all public domains.
|
||||
- Traefik can select the certificate and route by SNI/Host.
|
||||
- Certificate issuance and renewal are managed in one place.
|
||||
- The design is independent of the Keenetic wildcard certificate.
|
||||
- Swarm backends remain behind Traefik and are not exposed directly.
|
||||
|
||||
## Consequences
|
||||
|
||||
- WAN TCP `80` is a required dependency for every HTTP-01 validation and must
|
||||
reach Traefik directly.
|
||||
- ACME storage must be persistent across Traefik restarts and upgrades.
|
||||
- A valid email address must be supplied to the ACME resolver configuration.
|
||||
- Router-side changes are manual and are not represented by Kubernetes GitOps.
|
||||
- The first rollout must be verified at DNS, TCP, ACME, TLS, Ingress, and
|
||||
backend layers separately.
|
||||
|
||||
## Rejected alternatives
|
||||
|
||||
### TLS termination on Keenetic
|
||||
|
||||
Rejected because it makes the router's TLS proxy the certificate owner and was
|
||||
already observed to return the KeenDNS default `404` instead of reaching the
|
||||
Traefik host route.
|
||||
|
||||
### One public port per domain
|
||||
|
||||
Rejected for the standard URL requirement. It would require URLs such as
|
||||
`https://immich.rudolf.keenetic.link:8443/` and would not serve the default
|
||||
port `443`.
|
||||
|
||||
### DNS-01
|
||||
|
||||
Not selected because the requested operating model is HTTP-01. DNS-01 can be
|
||||
considered later if direct public port `80` cannot be provided.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `web` listens on TCP `80` and `websecure` listens on TCP `443`.
|
||||
- Keenetic forwards TCP `80` and `443` to `192.168.1.114` without TLS
|
||||
termination.
|
||||
- The ACME storage is persistent and not committed with credentials.
|
||||
- `curl -4I http://immich.rudolf.keenetic.link/` returns a permanent redirect to
|
||||
the equivalent `https://` URL.
|
||||
- HTTP-01 validation succeeds for every hostname requested in the Ingress.
|
||||
- `curl --resolve <host>:443:<wan-ip> https://<host>/` completes TLS with a
|
||||
certificate valid for `<host>` and reaches the expected backend.
|
||||
- The ACME challenge path remains available on `web` and does not depend on the
|
||||
KeenDNS web proxy.
|
||||
@@ -4,8 +4,15 @@ metadata:
|
||||
name: hermes-ui
|
||||
labels:
|
||||
app.kubernetes.io/name: hermes
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- hermes.rudolf.keenetic.link
|
||||
rules:
|
||||
- host: hermes.rudolf.keenetic.link
|
||||
http:
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Shared public ingress
|
||||
|
||||
The common Traefik entrypoint routes public domains to Kubernetes and Swarm
|
||||
backends. The router forwards TCP port `80` to `xu4_lan` (`192.168.1.114`).
|
||||
backends.
|
||||
|
||||
| Domain | Backend |
|
||||
| --- | --- |
|
||||
@@ -13,3 +13,116 @@ backends. The router forwards TCP port `80` to `xu4_lan` (`192.168.1.114`).
|
||||
The Swarm backends are represented by selectorless Kubernetes Services and
|
||||
static Endpoints. The Swarm ports must remain reachable from the Traefik node
|
||||
at `192.168.1.96`.
|
||||
|
||||
## HTTPS target configuration
|
||||
|
||||
The target design is TCP forwarding to Traefik, with TLS termination in
|
||||
Traefik:
|
||||
|
||||
```text
|
||||
WAN TCP 80 -> 192.168.1.114:80
|
||||
WAN TCP 443 -> 192.168.1.114:443
|
||||
```
|
||||
|
||||
Configure the Keenetic entries as raw TCP forwarding. Do not use the KeenDNS
|
||||
HTTPS proxy, terminate TLS on the router, rewrite SNI, or replace the HTTP
|
||||
`Host` header.
|
||||
|
||||
Traefik must provide:
|
||||
|
||||
- `web` on TCP `80` for HTTP and ACME HTTP-01;
|
||||
- `websecure` on TCP `443` for TLS termination;
|
||||
- a persistent ACME storage file, for example `/data/acme.json`;
|
||||
- an ACME resolver with `httpChallenge.entryPoint: web`;
|
||||
- a non-placeholder registration email supplied outside Git.
|
||||
|
||||
The exact `HelmChartConfig` values must match the installed K3s Traefik chart
|
||||
version. Inspect the live `HelmChart/traefik` values before applying a chart
|
||||
override; do not assume that a values schema from another K3s release is
|
||||
compatible.
|
||||
|
||||
For each public Ingress, pin the TLS route to `websecure`. The separate
|
||||
`http-redirect.yaml` resource attaches a `redirectScheme` middleware to
|
||||
`web`-only routes. Keeping the HTTP redirect separate prevents the internal
|
||||
ACME challenge router from being sent to a backend.
|
||||
|
||||
The resulting TLS Ingress shape is:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
|
||||
spec:
|
||||
tls:
|
||||
- hosts:
|
||||
- immich.rudolf.keenetic.link
|
||||
```
|
||||
|
||||
With a Traefik ACME resolver, the certificate is stored in the configured
|
||||
persistent ACME storage rather than committed to Git. If the installed chart
|
||||
requires a Kubernetes Secret for a different TLS mode, provision that Secret
|
||||
through the approved secret-management path. Do not commit `acme.json`, a
|
||||
private key, an API token, or an unencrypted Kubernetes Secret.
|
||||
|
||||
## Declarative traffic configuration
|
||||
|
||||
The repository configuration is in `traefik-https-config.yaml`. It extends the
|
||||
installed K3s `HelmChart/traefik` with:
|
||||
|
||||
- `web` exposed on TCP `80` for the HTTP redirect and ACME HTTP-01;
|
||||
- `websecure` exposed on TCP `443` for TLS termination;
|
||||
- a permanent HTTP-to-HTTPS `redirectScheme` middleware on web-only routes;
|
||||
- the `letsencrypt` ACME resolver using HTTP-01 through `web` (Traefik's
|
||||
ACME challenge handling remains available on the `web` entrypoint);
|
||||
- persistent `/data/acme.json` storage through a `1Gi` PVC.
|
||||
|
||||
The resolver uses the operator-supplied registration email from the
|
||||
configuration. The email is not a credential; the ACME account state and
|
||||
certificate private keys remain in persistent `/data/acme.json` storage and
|
||||
must not be committed to Git. The TLS annotations and host lists are present
|
||||
in both the shared public Ingress and Hermes-specific Ingress.
|
||||
|
||||
## Rollout procedure
|
||||
|
||||
1. Verify that every hostname resolves to the router's current public address.
|
||||
2. Configure Keenetic WAN TCP `80` and `443` forwarding to
|
||||
`192.168.1.114:80` and `192.168.1.114:443`.
|
||||
3. Confirm from an external network that TCP `80` reaches Traefik. The
|
||||
KeenDNS default `404` is not proof of correct forwarding.
|
||||
4. Configure the Traefik `websecure` entrypoint, persistent ACME storage, and
|
||||
HTTP-01 resolver through the repository's GitOps path.
|
||||
5. Add TLS annotations and host entries to the public Ingress resources.
|
||||
6. Apply through Flux only after rendering and reviewing the complete diff.
|
||||
7. Check the Traefik logs and ACME resource/state for the challenge result.
|
||||
8. Test HTTP-01 reachability:
|
||||
|
||||
```sh
|
||||
curl -4v http://immich.rudolf.keenetic.link/.well-known/acme-challenge/test
|
||||
```
|
||||
|
||||
This request must reach Traefik, not the KeenDNS web proxy.
|
||||
|
||||
9. Test TLS and SNI directly against the WAN address:
|
||||
|
||||
```sh
|
||||
curl -4vk \
|
||||
--resolve immich.rudolf.keenetic.link:443:<wan-ip> \
|
||||
https://immich.rudolf.keenetic.link/
|
||||
```
|
||||
|
||||
10. Verify the certificate SAN, the Traefik route, the HTTP status, and the
|
||||
Immich backend separately. Repeat the TLS/SNI test for every public
|
||||
hostname.
|
||||
|
||||
## Failure boundaries
|
||||
|
||||
- DNS resolves incorrectly: fix DNS before changing Kubernetes.
|
||||
- TCP `80` or `443` is refused: fix Keenetic forwarding or WAN reachability.
|
||||
- KeenDNS certificate/default `404` appears: traffic still uses the router
|
||||
proxy instead of TCP forwarding.
|
||||
- ACME challenge fails: inspect port `80`, Host preservation, and Traefik
|
||||
resolver/storage before changing backend routes.
|
||||
- TLS succeeds but the route is `404`/`502`: inspect the Ingress, Service,
|
||||
Endpoints, and Traefik access log separately.
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: Middleware
|
||||
metadata:
|
||||
name: http-to-https
|
||||
namespace: hermes
|
||||
spec:
|
||||
redirectScheme:
|
||||
scheme: https
|
||||
permanent: true
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: http-redirect
|
||||
namespace: hermes
|
||||
labels:
|
||||
app.kubernetes.io/name: http-redirect
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: web
|
||||
traefik.ingress.kubernetes.io/router.middlewares: hermes-http-to-https@kubernetescrd
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
rules:
|
||||
- host: hermes.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: hermes
|
||||
port:
|
||||
name: dashboard
|
||||
- host: brencher.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: brencher-swarm
|
||||
port:
|
||||
name: http
|
||||
- host: immich.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: immich-swarm
|
||||
port:
|
||||
name: http
|
||||
- host: gitea.rudolf.keenetic.link
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: gitea-swarm
|
||||
port:
|
||||
name: http
|
||||
@@ -2,4 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- swarm-backends.yaml
|
||||
- traefik-https-config.yaml
|
||||
- http-redirect.yaml
|
||||
- public-ingress.yaml
|
||||
|
||||
@@ -5,8 +5,17 @@ metadata:
|
||||
namespace: hermes
|
||||
labels:
|
||||
app.kubernetes.io/name: public-ingress
|
||||
annotations:
|
||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
|
||||
spec:
|
||||
ingressClassName: traefik
|
||||
tls:
|
||||
- hosts:
|
||||
- brencher.rudolf.keenetic.link
|
||||
- immich.rudolf.keenetic.link
|
||||
- gitea.rudolf.keenetic.link
|
||||
rules:
|
||||
- host: brencher.rudolf.keenetic.link
|
||||
http:
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChartConfig
|
||||
metadata:
|
||||
name: traefik
|
||||
namespace: kube-system
|
||||
spec:
|
||||
valuesContent: |-
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: xu4
|
||||
deployment:
|
||||
replicas: 1
|
||||
ports:
|
||||
web:
|
||||
port: 8000
|
||||
exposedPort: 80
|
||||
expose:
|
||||
default: true
|
||||
protocol: TCP
|
||||
websecure:
|
||||
port: 8443
|
||||
exposedPort: 443
|
||||
expose:
|
||||
default: true
|
||||
protocol: TCP
|
||||
additionalArguments:
|
||||
- "--entrypoints.web.address=:8000/tcp"
|
||||
- "--entrypoints.websecure.address=:8443/tcp"
|
||||
- "--certificatesresolvers.letsencrypt.acme.email=rudolfss13@gmail.com"
|
||||
- "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
|
||||
- "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
|
||||
persistence:
|
||||
enabled: true
|
||||
name: data
|
||||
path: /data
|
||||
accessMode: ReadWriteOnce
|
||||
size: 1Gi
|
||||
Reference in New Issue
Block a user