## Problem Let's Encrypt HTTP-01 requests were reaching ordinary Kubernetes HTTP routes instead of Traefik's internal ACME challenge handler. The CA observed backend responses (404/401/403), so Traefik continued serving its default certificate. ## Change - remove the Kubernetes `web` catch-all redirect Ingress and its backend routes; - restore Traefik entrypoint HTTP-to-HTTPS redirect; - set the redirect port explicitly to public `443`, preventing internal `:8443` in `Location`; - keep `web` available for the built-in HTTP-01 challenge router; - update the ingress decision/runbook. ## Verification - `git diff --check` passes; - `kubectl kustomize k3s_rencher/infrastructure/ingress` renders successfully; - rendered resources no longer include the Kubernetes HTTP redirect catch-all; - the Traefik redirect explicitly targets public port `443`. ## Post-merge acceptance Verify externally that HTTP redirects to `https://<host>/` without `:8443`, a real HTTP-01 attempt succeeds, and the served certificate has Let's Encrypt issuer/SANs for all four domains. --------- Co-authored-by: Hermes Agent <hermes@localhost> Reviewed-on: #23 Reviewed-by: admin <rudolfss13@gmail.com> Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
4.9 KiB
Shared public ingress
The common Traefik entrypoint routes public domains to Kubernetes and Swarm backends.
| Domain | Backend |
|---|---|
hermes.rudolf.keenetic.link |
Kubernetes Service hermes:9119 |
brencher.rudolf.keenetic.link |
Swarm 192.168.1.96:5001 |
immich.rudolf.keenetic.link |
Swarm 192.168.1.96:2283 |
gitea.rudolf.keenetic.link |
Swarm 192.168.1.96:3002 |
The Swarm backends are represented by selectorless Kubernetes Services and
static Endpoints. The Swarm ports must remain reachable from the Traefik node
at 192.168.1.96.
HTTPS target configuration
The target design is TCP forwarding to Traefik, with TLS termination in Traefik:
WAN TCP 80 -> 192.168.1.114:80
WAN TCP 443 -> 192.168.1.114:443
Configure the Keenetic entries as raw TCP forwarding. Do not use the KeenDNS
HTTPS proxy, terminate TLS on the router, rewrite SNI, or replace the HTTP
Host header.
Traefik must provide:
webon TCP80for HTTP and ACME HTTP-01;websecureon TCP443for TLS termination;- a persistent ACME storage file, for example
/data/acme.json; - an ACME resolver with
httpChallenge.entryPoint: web; - a non-placeholder registration email supplied outside Git.
The exact HelmChartConfig values must match the installed K3s Traefik chart
version. Inspect the live HelmChart/traefik values before applying a chart
override; do not assume that a values schema from another K3s release is
compatible.
For each public Ingress, pin the TLS route to websecure. Ordinary HTTP
requests are redirected by the Traefik web entrypoint to the public HTTPS
port 443; no Kubernetes HTTP catch-all route is used.
The resulting TLS Ingress shape is:
metadata:
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
spec:
tls:
- hosts:
- immich.rudolf.keenetic.link
With a Traefik ACME resolver, the certificate is stored in the configured
persistent ACME storage rather than committed to Git. If the installed chart
requires a Kubernetes Secret for a different TLS mode, provision that Secret
through the approved secret-management path. Do not commit acme.json, a
private key, an API token, or an unencrypted Kubernetes Secret.
Declarative traffic configuration
The repository configuration is in traefik-https-config.yaml. It extends the
installed K3s HelmChart/traefik with:
webexposed on TCP80for the HTTP redirect and ACME HTTP-01;websecureexposed on TCP443for TLS termination;- a permanent HTTP-to-HTTPS entrypoint redirect with public port
443; - the
letsencryptACME resolver using HTTP-01 throughweb(Traefik's ACME challenge handling remains available on thewebentrypoint); - persistent
/data/acme.jsonstorage through a1GiPVC.
The resolver uses the operator-supplied registration email from the
configuration. The email is not a credential; the ACME account state and
certificate private keys remain in persistent /data/acme.json storage and
must not be committed to Git. The TLS annotations and host lists are present
in both the shared public Ingress and Hermes-specific Ingress.
Rollout procedure
-
Verify that every hostname resolves to the router's current public address.
-
Configure Keenetic WAN TCP
80and443forwarding to192.168.1.114:80and192.168.1.114:443. -
Confirm from an external network that TCP
80reaches Traefik. The KeenDNS default404is not proof of correct forwarding. -
Configure the Traefik
websecureentrypoint, persistent ACME storage, and HTTP-01 resolver through the repository's GitOps path. -
Add TLS annotations and host entries to the public Ingress resources.
-
Apply through Flux only after rendering and reviewing the complete diff.
-
Check the Traefik logs and ACME resource/state for the challenge result.
-
Test HTTP-01 reachability:
curl -4v http://immich.rudolf.keenetic.link/.well-known/acme-challenge/testThis request must reach Traefik, not the KeenDNS web proxy.
-
Test TLS and SNI directly against the WAN address:
curl -4vk \ --resolve immich.rudolf.keenetic.link:443:<wan-ip> \ https://immich.rudolf.keenetic.link/ -
Verify the certificate SAN, the Traefik route, the HTTP status, and the Immich backend separately. Repeat the TLS/SNI test for every public hostname.
Failure boundaries
- DNS resolves incorrectly: fix DNS before changing Kubernetes.
- TCP
80or443is refused: fix Keenetic forwarding or WAN reachability. - KeenDNS certificate/default
404appears: traffic still uses the router proxy instead of TCP forwarding. - ACME challenge fails: inspect port
80, Host preservation, and Traefik resolver/storage before changing backend routes. - TLS succeeds but the route is
404/502: inspect the Ingress, Service, Endpoints, and Traefik access log separately.