Files
hermesandHermes Agent 0d29c7d7af fix(ingress): let Traefik own HTTP-01 challenge path (#23)
## Problem

Let's Encrypt HTTP-01 requests were reaching ordinary Kubernetes HTTP routes instead of Traefik's internal ACME challenge handler. The CA observed backend responses (404/401/403), so Traefik continued serving its default certificate.

## Change

- remove the Kubernetes `web` catch-all redirect Ingress and its backend routes;
- restore Traefik entrypoint HTTP-to-HTTPS redirect;
- set the redirect port explicitly to public `443`, preventing internal `:8443` in `Location`;
- keep `web` available for the built-in HTTP-01 challenge router;
- update the ingress decision/runbook.

## Verification

- `git diff --check` passes;
- `kubectl kustomize k3s_rencher/infrastructure/ingress` renders successfully;
- rendered resources no longer include the Kubernetes HTTP redirect catch-all;
- the Traefik redirect explicitly targets public port `443`.

## Post-merge acceptance

Verify externally that HTTP redirects to `https://<host>/` without `:8443`, a real HTTP-01 attempt succeeds, and the served certificate has Let's Encrypt issuer/SANs for all four domains.

---------

Co-authored-by: Hermes Agent <hermes@localhost>
Reviewed-on: #23
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-22 22:29:19 +01:00

4.9 KiB

Shared public ingress

The common Traefik entrypoint routes public domains to Kubernetes and Swarm backends.

Domain Backend
hermes.rudolf.keenetic.link Kubernetes Service hermes:9119
brencher.rudolf.keenetic.link Swarm 192.168.1.96:5001
immich.rudolf.keenetic.link Swarm 192.168.1.96:2283
gitea.rudolf.keenetic.link Swarm 192.168.1.96:3002

The Swarm backends are represented by selectorless Kubernetes Services and static Endpoints. The Swarm ports must remain reachable from the Traefik node at 192.168.1.96.

HTTPS target configuration

The target design is TCP forwarding to Traefik, with TLS termination in Traefik:

WAN TCP 80  -> 192.168.1.114:80
WAN TCP 443 -> 192.168.1.114:443

Configure the Keenetic entries as raw TCP forwarding. Do not use the KeenDNS HTTPS proxy, terminate TLS on the router, rewrite SNI, or replace the HTTP Host header.

Traefik must provide:

  • web on TCP 80 for HTTP and ACME HTTP-01;
  • websecure on TCP 443 for TLS termination;
  • a persistent ACME storage file, for example /data/acme.json;
  • an ACME resolver with httpChallenge.entryPoint: web;
  • a non-placeholder registration email supplied outside Git.

The exact HelmChartConfig values must match the installed K3s Traefik chart version. Inspect the live HelmChart/traefik values before applying a chart override; do not assume that a values schema from another K3s release is compatible.

For each public Ingress, pin the TLS route to websecure. Ordinary HTTP requests are redirected by the Traefik web entrypoint to the public HTTPS port 443; no Kubernetes HTTP catch-all route is used.

The resulting TLS Ingress shape is:

metadata:
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
spec:
  tls:
    - hosts:
        - immich.rudolf.keenetic.link

With a Traefik ACME resolver, the certificate is stored in the configured persistent ACME storage rather than committed to Git. If the installed chart requires a Kubernetes Secret for a different TLS mode, provision that Secret through the approved secret-management path. Do not commit acme.json, a private key, an API token, or an unencrypted Kubernetes Secret.

Declarative traffic configuration

The repository configuration is in traefik-https-config.yaml. It extends the installed K3s HelmChart/traefik with:

  • web exposed on TCP 80 for the HTTP redirect and ACME HTTP-01;
  • websecure exposed on TCP 443 for TLS termination;
  • a permanent HTTP-to-HTTPS entrypoint redirect with public port 443;
  • the letsencrypt ACME resolver using HTTP-01 through web (Traefik's ACME challenge handling remains available on the web entrypoint);
  • persistent /data/acme.json storage through a 1Gi PVC.

The resolver uses the operator-supplied registration email from the configuration. The email is not a credential; the ACME account state and certificate private keys remain in persistent /data/acme.json storage and must not be committed to Git. The TLS annotations and host lists are present in both the shared public Ingress and Hermes-specific Ingress.

Rollout procedure

  1. Verify that every hostname resolves to the router's current public address.

  2. Configure Keenetic WAN TCP 80 and 443 forwarding to 192.168.1.114:80 and 192.168.1.114:443.

  3. Confirm from an external network that TCP 80 reaches Traefik. The KeenDNS default 404 is not proof of correct forwarding.

  4. Configure the Traefik websecure entrypoint, persistent ACME storage, and HTTP-01 resolver through the repository's GitOps path.

  5. Add TLS annotations and host entries to the public Ingress resources.

  6. Apply through Flux only after rendering and reviewing the complete diff.

  7. Check the Traefik logs and ACME resource/state for the challenge result.

  8. Test HTTP-01 reachability:

    curl -4v http://immich.rudolf.keenetic.link/.well-known/acme-challenge/test
    

    This request must reach Traefik, not the KeenDNS web proxy.

  9. Test TLS and SNI directly against the WAN address:

    curl -4vk \
      --resolve immich.rudolf.keenetic.link:443:<wan-ip> \
      https://immich.rudolf.keenetic.link/
    
  10. Verify the certificate SAN, the Traefik route, the HTTP status, and the Immich backend separately. Repeat the TLS/SNI test for every public hostname.

Failure boundaries

  • DNS resolves incorrectly: fix DNS before changing Kubernetes.
  • TCP 80 or 443 is refused: fix Keenetic forwarding or WAN reachability.
  • KeenDNS certificate/default 404 appears: traffic still uses the router proxy instead of TCP forwarding.
  • ACME challenge fails: inspect port 80, Host preservation, and Traefik resolver/storage before changing backend routes.
  • TLS succeeds but the route is 404/502: inspect the Ingress, Service, Endpoints, and Traefik access log separately.