Author SHA1 Message Date
hermesandHermes Agent 0f9c5fa105 fix(ingress): isolate HTTP redirect from ACME (#22)
See commit 1af4832. Removes the broken global redirect that exposed internal :8443, pins TLS routes to websecure, and adds a dedicated web redirect middleware so ACME HTTP-01 remains isolated.

---------

Co-authored-by: Hermes Agent <hermes@localhost>
Reviewed-on: org1/infra#22
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-22 17:49:41 +01:00
hermesandHermes Agent 24198a56e4 [in-review] feat(ingress): Traefik TLS on 443 with HTTP-01 (#21)
## Summary
- add the declarative K3s `HelmChartConfig` for Traefik `web`/`websecure`
- expose TCP 80/443 and configure the `letsencrypt` HTTP-01 resolver
- persist `/data/acme.json` through a PVC
- add TLS routing annotations and host lists for the shared public Ingress and Hermes Ingress
- document the router cutover, verification procedure, and separate DEC-012 decision

## Scope
This PR contains the traffic configuration and documentation. It does not execute `kubectl apply`, change Keenetic, issue certificates, or commit credentials.

## Required operator input before apply
- direct WAN TCP 80 and 443 forwarding to 192.168.1.114
- the real Let's Encrypt registration email, inserted through the approved values/secret workflow rather than invented in Git
- final review of the installed Traefik chart values and PVC/storage behavior

## Verification
- `git diff --check` passed
- base: `e084ce4ecf39f19d08ec97fee852f59310e0cdbb`
- head: `4f70885`

Related decision: `docs/decisions/2026-11-21-traefik-tls-http01.md`

---------

Co-authored-by: Hermes Agent <hermes@localhost>
Reviewed-on: org1/infra#21
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-22 17:31:10 +01:00
hermes e084ce4ecf feat(ingress): route public domains to Swarm backends (#14)
## Summary

Make Kubernetes Traefik the common HTTP entrypoint while preserving the existing Swarm-hosted public domains.

## Routing

- `hermes.rudolf.keenetic.link` -> Kubernetes Service `hermes:9119`
- `brencher.rudolf.keenetic.link` -> `192.168.1.96:5001`
- `immich.rudolf.keenetic.link` -> `192.168.1.96:2283`
- `gitea.rudolf.keenetic.link` -> `192.168.1.96:3002`

The Swarm targets are represented by selectorless Services with static Endpoints. The router can use ordinary TCP forwarding to `192.168.1.114:80`, preserving the HTTP `Host` header for Traefik routing instead of using the Keenetic HTTP reverse-proxy feature.

## Validation

- `git diff --cached --check` passed
- Changed files are limited to the Hermes ingress, backend Services/Endpoints, Kustomization, and README
- `kubectl kustomize` was unavailable in the execution environment

Reviewed-on: org1/infra#14
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-21 20:36:59 +01:00
hermesandhermes 696aacfabe docs(k3s): configure node-local registry DNS at join (#19)
## Summary

- document a one-command `rudolf-T` bootstrap
- mount a dedicated resolver into `k3s-agent` only
- keep the Ubuntu host resolver unchanged for other services

The resolver is used by embedded containerd during image pulls. The registry hostname and TLS verification remain unchanged.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: org1/infra#19
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-21 20:07:40 +01:00
hermesandhermes c0e042f0a0 fix(k3s): use valid generic namespace names (#18)
Flux reconciliation of PR #17 failed because `hermes_generic1` and `hermes_generic2` are invalid Kubernetes namespace names.

This PR renames them to `hermes-generic1` and `hermes-generic2` in both Namespace manifests and Kustomize overlays.

Validation:
- main render remains byte-identical to admin/main
- generic1, generic2, and root renders succeed with Kustomize v5.8.3
- git diff --check passes
- no underscore namespace names remain

Fixes the Flux dry-run error reported by the live cluster.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: org1/infra#18
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-21 13:56:04 +01:00
hermesandhermes 95edba82bf [ready-to-test] refactor(k3s): base Hermes infra and add generic overlay (#17)
Replace the duplicated hermes/hermes_generic manifests with a shared Kubernetes base derived from legacy hermes_infra. Add hermes_infra and generic overlays with separate namespaces, NFS PV/PVC state, and credentials. Preserve the external Swarm docker socket through a namespaced Service/Endpoints because K3s uses containerd and has no local Docker socket. Sarg is intentionally deferred until its dynamically tagged image has a stable Kubernetes reference.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: org1/infra#17
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-21 13:20:45 +01:00
hermesandhermes 5e314e17d0 fix(hermes): remove unused FreeCAD MCP socket port (#16)
neka uses XML-RPC on TCP 9875. Remove the unused TCP 9876 egress rule.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: org1/infra#16
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-19 18:05:41 +01:00
hermes 5e45811200 feat(hermes): allow FreeCAD MCP egress (#15)
Allow the Kubernetes Hermes pod to reach the FreeCAD MCP bridge on rudolf-T at 100.105.201.15.

- TCP 9875: XML-RPC
- TCP 9876: bridge socket
- exact destination /32 only

This changes the NetworkPolicy only; live connectivity still requires the PR to be merged and reconciled by Flux.

Reviewed-on: org1/infra#15
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-19 15:56:57 +01:00
hermes 6a7ad7e235 fix(hermes): reuse existing ingress resource name (#13)
The merged public Ingress was created as `hermes`, while a legacy `hermes-ui` Ingress with the same host remained in the cluster outside the current Flux inventory. This creates two routes for the same host.

Reuse the existing `hermes-ui` resource name so Flux updates it in place and prunes the inventory-managed `hermes` resource, leaving one route for `hermes.rudolf.keenetic.link`.

Reviewed-on: org1/infra#13
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-18 23:40:13 +01:00
hermes 2c84082e22 feat(hermes): expose dashboard through public ingress (#12)
## What changed

- replace the broken in-cluster Nginx UI deployment with a Traefik Ingress for `hermes.rudolf.keenetic.link`;
- route the Ingress to the existing `hermes` Service on its named `dashboard` port (`9119`);
- allow only the live Traefik pods in `kube-system` to reach Hermes port `9119`;
- remove the UI Nginx capability failure path (`setgid(101) Operation not permitted`);
- remove the stale UI middleware/sidecar definition from the Hermes kustomization so Flux can prune it;
- document the router mapping and verification path.

## Root cause observed in the cluster

The live UI returned `502 Bad Gateway`. The deployed UI sidecar had Nginx workers failing with `setgid(101) Operation not permitted`, while the Ingress was targeting the sidecar path and referenced a middleware that Traefik could not resolve. The direct application Service already exposes the dashboard and WebSocket endpoint on port `9119`, so the sidecar is unnecessary.

## Router

Keep the public mapping:

`hermes.rudolf.keenetic.link` → `xu4_lan` (`192.168.1.114`) → HTTP `80`.

Do not forward to the Kubernetes Service port `9119` or to a Tailscale address from the router. The router forwards to the Kubernetes ingress entrypoint; the Kubernetes Ingress selects the `hermes` Service.

## Verification

- base branch is current `org1/infra:main`;
- `git diff --check` passes;
- no conflict markers are present in tracked files;
- live rollout verification remains pending until Flux reconciles this PR, after which verify the Ingress, Traefik route, Service endpoints, and external URL.

Reviewed-on: org1/infra#12
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-18 23:34:55 +01:00
hermes 7a029d5c85 docs: record Keenetic remote access domains (#11)
Add DEC-009 to record the current remote web application domain management model.

Recorded facts:
- router model: Keenetic Hero 4G;
- brencher.rudolf.keenetic.link: authorized access;
- immich.rudolf.keenetic.link: public access;
- gitea.rudolf.keenetic.link: no access;
- hermes.rudolf.keenetic.link: public access.

The decision explicitly distinguishes HTTP/HTTPS web application publishing from arbitrary TCP port forwarding for Docker API ports 2375 and 2376. No image is added.

Reviewed-on: org1/infra#11
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-18 20:16:10 +01:00
hermes 410a0be8e3 feat(k3s): allow Hermes access to Docker Swarm proxy (#10)
Allow the Kubernetes Hermes pod to reach the Docker Swarm socket proxy used for read-only Swarm inspection.

The session history shows the established endpoint as `192.168.1.96:2375`; previous direct checks used Docker API `/v1.44/version`, `/version`, `/_ping`, `/services`, and `/volumes`.

This PR adds only a narrow egress rule for TCP 2375 to `192.168.1.96/32`. It does not add an environment variable or change the socket proxy's read-only configuration.

Reviewed-on: org1/infra#10
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-17 15:42:22 +01:00
admin 4e1dfc905f x 2026-11-17 11:17:02 +01:00
admin 5ba8ae21e0 x 2026-11-17 11:14:29 +01:00
admin 5b5ede0b65 ч 2026-11-16 22:17:36 +01:00
hermes b0c5413bf7 fix(k3s): use NFSv3 for Hermes state volume (#5)
The merged Hermes NFS PV used NFSv4, but the existing NFS server export is reachable with NFSv3, matching the Swarm volume configuration. This fixes the mount access denied error during rollout.

Reviewed-on: org1/infra#5
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 20:57:57 +01:00
hermes 9a0551192a feat(k3s): move Hermes state to adjacent NFS path (#4)
Use a dedicated static NFS-backed PV/PVC for the Kubernetes Hermes state at /root/volumes/hermes/k3s. Keep the existing local-path PVC declared for rollback and switch the Deployment to the new claim.

The path is separate from the Swarm Hermes gateway and agent state directories. The NFS directory must be created on 192.168.1.96 before rollout.

Reviewed-on: org1/infra#4
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-16 20:39:02 +01:00
hermesandhermes e1a7faf1fb fix(k3s): reconcile Flux from infra repository (#3)
## Summary
Move Flux K3s reconciliation from the removed `org1/k3s` repository to `org1/infra`.

## Changes
- point the `k3s` GitRepository at `org1/infra.git`
- reconcile the `./k3s_rencher` path
- update the GitOps documentation

## Verification
- `git diff --check`
- reviewed the repository tree and confirmed `org1/k3s` is absent

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: org1/infra#3
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
2026-11-14 22:27:19 +01:00
admin f4c54f8010 x 2026-11-13 20:53:47 +01:00
admin 5cb4642d67 hermes 2026-11-13 20:24:30 +01:00
admin 2889600de1 x 2026-11-12 23:08:52 +01:00
admin 7789e95247 x 2026-11-11 13:45:05 +01:00
admin ca63dd6069 x 2026-11-11 13:42:00 +01:00
hermesandadmin fc6f668a86 [ready-to-test] k3s: expand agent read access without secrets (#1)
## Summary

Expand the `agent-gateway/agent-readonly` service account from the built-in `view` role to a dedicated read-only role.

## Access policy

- Adds `get`, `list`, and `watch` for the core cluster resources and all currently served non-core API groups.
- Adds read-only non-resource access for API discovery, health, OpenAPI, and version endpoints.
- Does not grant access to the Kubernetes `secrets` resource.
- Does not grant write verbs, exec, port-forward, proxy, or token creation permissions.

## Verification

- Current target `org1/k3s:main`: `2bbac8467a9297e7056956328f576978142f3b24`
- `git diff --check`: passed
- Changed file: `k3s_rencher/infrastructure/agent-gateway/rbac.yaml`

The existing binding name is preserved so Flux applies the role change in place.

## Kustomize inclusion

The root `k3s_rencher/kustomization.yaml` already includes `infrastructure/agent-gateway`. Its nested Kustomization includes `rbac.yaml`, so the new ClusterRole is applied transitively while preserving the `agent-gateway` namespace transformation. A duplicate direct root entry would cause duplicate resource processing.

---------

Co-authored-by: admin <rudolfss13@gmail.com>
Reviewed-on: org1/k3s#1
Co-authored-by: Alex Rudenk <rudolfss13+hermes@gmail.com>
2026-11-11 13:02:40 +01:00
42 changed files with 1884 additions and 15 deletions
-5
View File
@@ -1,5 +0,0 @@
# Project Instructions
## Docker Swarm Stack Files
All YAML files in this project are Docker Swarm stack files. Keep them compatible with Docker Swarm and deploy them with `docker stack deploy -c <file> <stack-name>`, not `docker compose up`.
+88
View File
@@ -0,0 +1,88 @@
# Repository Decisions
This file records repository decisions and the guidance that follows from them. Accepted decisions are in force. Draft decisions are proposals, not policy. Declined decisions are retained for context and are not in force.
## Accepted
### DEC-001: Make the Kubernetes cluster the primary infrastructure
- **Decision:** New services should be deployed in the Kubernetes cluster, and services running on other instances should be migrated there.
[kustomization.yaml](kustomization.yaml) is entry point for the Kubernetes cluster configuration.
### DEC-002: Document manual operations in README.md
- **Decision:** The repository's `README.md` should document useful manual operations and link to all other manual instructions in the repository.
### DEC-003: Retire Ansible
- **Decision:** Ansible is legacy and should be removed strategically.
### DEC-004: Treat Docker Swarm files as legacy
- **Decision:** Docker Swarm files are legacy. New services should not use Docker Swarm and should adopt Kubernetes or other modern orchestration solutions.
### DEC-005: Isolate Kubernetes namespaces by default
- **Decision:** Each namespace should be isolated by default for ingress and egress. Egress should allow only explicitly whitelisted connectivity.
### DEC-006: Use GitOps for infrastructure changes
- **Decision:** Use GitOps as much as possible and keep the repository as the source for infrastructure. Automate changes through CI/CD. Make infrastructure changes through pull requests and have them reviewed before application.
### DEC-007: Preserve Docker Swarm deployment for legacy stack files
- **Decision:** Legacy Docker Swarm stack files should remain compatible with Docker Swarm and be deployed with `docker stack deploy -c <file> <stack-name>`, not `docker compose up`.
### DEC-008: Apply configuration through Git pushes only
- **Decision:** Configuration should be applied only through Git pushes. Do not apply configuration directly to infrastructure, even when access is available. An agent may apply configs directly tom infrastructure only when explicitly requested.
### DEC-009: Use separate NFS state for Kubernetes Hermes
- **Context:** Kubernetes Hermes must not share mutable state with the legacy Swarm Hermes instance. K3s `local-path` storage is node-local and does not provide the desired storage layout.
- **Decision:** Kubernetes Hermes uses a static NFS-backed `hermes-data-nfs` claim from `192.168.1.96:/root/volumes/hermes/k3s`, mounted at `/opt/data`. The volume uses NFSv3 and `Retain` reclaim policy. The Swarm directories `/root/volumes/hermes/gateway` and `/root/volumes/hermes/agent` remain separate.
- **Consequences:** The NFS export must allow the current Kubernetes node addresses `100.70.193.97`, `100.105.201.15`, and `100.96.92.102`, in addition to the existing Swarm LAN allowance `192.168.1.0/24`. Kubernetes and Swarm Hermes must not use the same mutable state directory concurrently.
### DEC-010: Manage web application domains through Keenetic Hero 4G
- **Context:** Remote access to web applications in the network is managed through the Keenetic Hero 4G router's web application access settings.
- **Decision:** The router-managed web application mappings are:
| Domain | Remote access | Host | IP address | Protocol | TCP port |
| --- | --- | --- | --- | --- | ---: |
| `brencher.rudolf.keenetic.link` | Authorized | `odroid` | `192.168.1.96` | HTTP | `5001` |
| `immich.rudolf.keenetic.link` | Public | `odroid` | `192.168.1.96` | HTTP | `2283` |
| `gitea.rudolf.keenetic.link` | No access | `odroid` | `192.168.1.96` | HTTP | `3002` |
| `hermes.rudolf.keenetic.link` | Public | `xu4_lan` | `192.168.1.114` | HTTP | `80` |
- **Consequences:** These entries describe HTTP/HTTPS web application publishing through the router. They do not imply that arbitrary TCP ports, including Docker API ports `2375` or `2376`, are published. Raw TCP access requires a separate router port-forward or another explicitly configured network path.
## Draft
### DEC-011: Add authenticated NFS access for Hermes
- **Context:** The current NFS access model relies on export client allowlists and Unix filesystem permissions. It does not provide NFS username/password authentication.
- **Decision:** Defer NFS authentication. Do not add Kerberos or store NFS credentials in Kubernetes as part of the current Hermes storage migration.
- **Open options:** Narrow exports to exact client addresses, replace `no_root_squash` with `root_squash` after validating Hermes UID/GID behavior, or evaluate Kerberos-backed NFS with `sec=krb5`/`sec=krb5p` in a separate change.
- **Status:** Draft; the current IP-restricted export model remains in use.
### DEC-012: Terminate public TLS in Traefik
- **Context:** The public Kubernetes Ingress routes HTTP traffic through Traefik, while the KeenDNS/Keenetic HTTPS path does not reliably preserve the route to the requested host. Standard HTTPS URLs must work for the public domains without per-domain port suffixes.
- **Decision:** Forward WAN TCP `80` and `443` directly to `192.168.1.114` and terminate TLS in Traefik. Use Traefik's Let's Encrypt HTTP-01 resolver on the `web` entrypoint and the `websecure` entrypoint on `443`. Keenetic must not terminate TLS or rewrite SNI/Host.
- **Consequences:** Public port `80` is required for certificate issuance and renewal; ordinary HTTP requests are redirected to HTTPS. Traefik ACME storage must be persistent. Router changes remain manual; the operational procedure is documented in [`k3s_rencher/infrastructure/ingress/README.md`](k3s_rencher/infrastructure/ingress/README.md).
- **Status:** Proposed; apply only after the router forwarding, ACME email, persistent storage, and live Traefik chart configuration are verified.
## Declined
No declined decisions recorded.
## Adding a decision
Add each decision under its current status. Use a unique `DEC-NNN` identifier and include the context, decision, and consequences. Drafts are not instructions; move a decision to Accepted when approved, or to Declined if it is not adopted.
### DEC-NNN: Short title
- **Context:** Why a decision is needed.
- **Decision:** What is decided.
- **Consequences:** Important effects or tradeoffs.
+161
View File
@@ -0,0 +1,161 @@
# Open findings
This file tracks findings from the repository compliance review against the
accepted decisions in [AGENTS.md](./AGENTS.md). The root
[README.md](./README.md) now documents the repository and manual operations;
the findings below are the remaining unresolved items.
## DEC-001 — Kubernetes is not yet the primary infrastructure
**Status:** Open
The root [kustomization.yaml](./kustomization.yaml) currently reconciles Flux,
DNS, the agent gateway, and network-policy scaffolding, but no production
application workloads. Production services are still documented as Docker
Swarm deployments in [PROD.md](./PROD.md), and service definitions remain
under [poc/](./poc/), [nextcloud/](./nextcloud/), and
[hermes_squid/](./hermes_squid/).
Required follow-up:
- Define target Kubernetes namespaces and ownership for production services.
- Add application overlays under `k3s_rencher/`.
- Migrate services to Kubernetes one at a time.
- Add migrated workloads to the root Kustomize entry point.
- Replace production Swarm instructions after each successful migration.
## DEC-003 — Ansible has not been retired
**Status:** Open
The repository still contains active-looking playbooks, inventory, variables,
Galaxy dependencies, and host-management roles:
- [all.yaml](./all.yaml)
- [gogs.yaml](./gogs.yaml)
- [orientdb.yaml](./orientdb.yaml)
- [syncthing.yaml](./syncthing.yaml)
- [hosts](./hosts)
- [group_vars/all](./group_vars/all)
- [requirements.yaml](./requirements.yaml)
- [roles/](./roles/)
Required follow-up:
- Inventory all Ansible-managed resources.
- Classify each resource as migrated, required for host bootstrap, obsolete,
or requiring a Kubernetes replacement.
- Retain only unavoidable host bootstrap.
- Deprecate and then remove obsolete playbooks, roles, inventory, and
dependencies after replacement procedures are documented.
## DEC-004 — Swarm remains the production deployment path
**Status:** Open
Legacy Swarm files may remain during migration, but [PROD.md](./PROD.md) still
instructs operators to initialize Swarm and deploy production services with
`docker stack deploy`. Service documentation also presents Swarm or Docker
Compose as the primary deployment method.
Relevant legacy material includes:
- [docker-compose-nginx.yaml](./docker-compose-nginx.yaml)
- [docker-compose-registry.yaml](./docker-compose-registry.yaml)
- [nextcloud/stack-compose.yaml](./nextcloud/stack-compose.yaml)
- [hermes_squid/stack-compose.yml](./hermes_squid/stack-compose.yml)
- [poc/immich/stack-compose.yml](./poc/immich/stack-compose.yml)
- [poc/gitea/stack-compose.yml](./poc/gitea/stack-compose.yml)
- [poc/vscode/stack-code-server.yml](./poc/vscode/stack-code-server.yml)
Required follow-up:
- Mark remaining Swarm files and instructions as legacy and identify their
migration status.
- Do not add new Swarm services.
- Migrate production services to Kubernetes and replace their operational
instructions.
- Keep legacy stack files compatible with `docker stack deploy` until removed.
## DEC-005 — Namespace isolation is incomplete
**Status:** Partially implemented
The default policies in
[default-deny.yaml](./k3s_rencher/infrastructure/network-policies/default-deny.yaml)
and the DNS exception in
[allow-dns.yaml](./k3s_rencher/infrastructure/network-policies/allow-dns.yaml)
are currently applied through the `agent-gateway` overlay only.
Required follow-up:
- Create a dedicated Kustomize overlay for every workload namespace.
- Apply default-deny ingress and egress to every workload namespace.
- Add explicit ingress, pod-to-pod, and external egress policies per service.
- Decide and document the policy baseline for `flux-system` and other system
namespaces.
- Verify that the externally sourced connectivity tester has equivalent
namespace and network-policy controls.
- Review all future uses of `hostNetwork`, host ports, privileged containers,
and node-originated traffic.
## DEC-006 — GitOps changes are not sufficiently validated or gated
**Status:** Partially implemented
Flux reconciliation is configured through
[flux-system/gitrepository.yaml](./k3s_rencher/flux-system/gitrepository.yaml)
and [flux-system/sync.yaml](./k3s_rencher/flux-system/sync.yaml), but the
repository has no visible pull-request validation workflow for Kubernetes
changes. The existing
[release workflow](./.gitea/workflows/release.yml) only builds and publishes
the connectivity tester image.
Required follow-up:
- Add pull-request checks for YAML, Kustomize rendering, Kubernetes schemas,
network policies, secrets, and image references.
- Protect the Flux-tracked branch and require review plus passing checks.
- Document the one-time bootstrap boundary versus normal Flux-managed changes.
- Add rollback and reconciliation verification procedures.
- Review whether public Git sources are an intentional trust decision.
- Prefer immutable image tags or digests over `latest` for production.
## Secret and credential hygiene
**Status:** Open
Potential credentials and unsafe examples remain in tracked files, including
[roles/metricbeat/defaults/main.yml](./roles/metricbeat/defaults/main.yml),
[roles/orientdb/defaults/main.yml](./roles/orientdb/defaults/main.yml),
[scratches/tomcat/tomcat-users.xml](./scratches/tomcat/tomcat-users.xml),
and [hermes_squid/README.md](./hermes_squid/README.md).
Required follow-up:
- Determine whether each value was ever active.
- Rotate any potentially active credential.
- Remove credentials from tracked files and examples.
- Use SOPS, External Secrets, or another approved secret manager for
Kubernetes workloads.
- Keep only clearly fake placeholders in documentation.
## Repository structure and lifecycle
**Status:** Open
The repository is still structured primarily as an Ansible project even though
the accepted direction is Kubernetes and GitOps. Kubernetes resources,
experimental deployments, legacy stacks, and historical scratch files are
mixed together.
Required follow-up:
- Establish clear ownership boundaries for cluster bootstrap,
infrastructure, applications, legacy deployments, and historical material.
- Consider separating `cluster-bootstrap/`, `infrastructure/`, `apps/`, and
`legacy/`.
- Archive or remove obsolete material only after its replacement and rollback
procedure are documented.
- Keep the root [README.md](./README.md) and this file updated as migrations
complete.
+46
View File
@@ -0,0 +1,46 @@
# Infrastructure repository
See [FINDINGS.md](./FINDINGS.md) for unresolved compliance findings and
required follow-up actions.
## Manual operations and service documentation
### Cluster and production operations
- [K3s installation and health operations](./k3s_rencher/README.md)
- [GitOps and Flux bootstrap](./k3s_rencher/README-gitops.md)
- [Legacy production setup](./PROD.md)
- [Legacy Docker Compose operations](./README_docker_compose.md)
### Service documentation
- [Nextcloud operations](./nextcloud/README.md)
- [Hermes Squid operations](./hermes_squid/README.md)
- [Anywherelan deployment](./poc/anywherelan/README.md)
- [Beszel stack](./poc/beszel/stack-compose.yml)
- [Gitea runner stack](./poc/gitea/stack-compose.yml)
- [Hermes stack](./poc/hermes/README.md)
- [Hermes Agent Kubernetes deployment](./k3s_rencher/apps/hermes/README.md)
- [Shared Traefik ingress and HTTPS operations](./k3s_rencher/infrastructure/ingress/README.md)
- [Traefik TLS/HTTP-01 decision](./docs/decisions/2026-11-21-traefik-tls-http01.md)
- [Immich stack](./poc/immich/README.md)
- [Lampa stack](./poc/lampa/README.md)
- [NetBird deployment](./poc/netbird/README.md)
- [VS Code server stack](./poc/vscode/README.md)
The service-specific Docker documentation remains valid only for services
that have not yet been migrated. Update the migration status above and the
service documentation when a service moves to Kubernetes.
## Repository layout
- [k3s_rencher/](./k3s_rencher/) — Kubernetes and Flux resources.
- [connectivity_tester/](./connectivity_tester/) — connectivity tester image
and tests.
- [poc/](./poc/) — legacy and experimental service deployments.
- [nextcloud/](./nextcloud/) and [hermes_squid/](./hermes_squid/) — legacy
service deployments.
- [roles/](./roles/) — legacy Ansible roles retained during migration.
- [scratches/](./scratches/) — historical experiments; not production
configuration.
@@ -0,0 +1,92 @@
# Decision: TLS termination in Traefik on TCP 443
- Status: proposed
- Date: 2026-11-21
- Scope: K3s edge ingress for public domains
## Context
The public HTTP routes are handled by the bundled Traefik instance on `xu4`
(`192.168.1.114`). The current public Ingress has HTTP rules for the Hermes,
Brencher, Immich, and Gitea hostnames. HTTPS is currently terminated by the
KeenDNS/Keenetic path and does not reliably reach the Traefik host rules.
The required target is standard HTTPS with no port suffix:
```text
https://immich.rudolf.keenetic.link/
```
The same Traefik listener must support SNI/Host-based routing for the other
public hostnames.
## Decision
Use TCP forwarding from Keenetic to the Traefik node and terminate TLS in
Traefik:
```text
WAN TCP 80 -> 192.168.1.114:80
WAN TCP 443 -> 192.168.1.114:443
```
Configure a Traefik `websecure` entrypoint on port `443` and use Traefik's
Let's Encrypt ACME resolver with the HTTP-01 challenge on the `web` entrypoint
(port `80`). Ordinary HTTP routes use a separate permanent `redirectScheme`
middleware on `web`; TLS routes are pinned to `websecure`. This keeps the
internal ACME challenge router separate from backend routes. Public Kubernetes
Ingress resources will opt into TLS using the same resolver.
The router must not terminate TLS, use a KeenDNS HTTPS proxy, rewrite SNI, or
replace the HTTP `Host` header on the forwarded path.
## Rationale
- One standard HTTPS port supports all public domains.
- Traefik can select the certificate and route by SNI/Host.
- Certificate issuance and renewal are managed in one place.
- The design is independent of the Keenetic wildcard certificate.
- Swarm backends remain behind Traefik and are not exposed directly.
## Consequences
- WAN TCP `80` is a required dependency for every HTTP-01 validation and must
reach Traefik directly.
- ACME storage must be persistent across Traefik restarts and upgrades.
- A valid email address must be supplied to the ACME resolver configuration.
- Router-side changes are manual and are not represented by Kubernetes GitOps.
- The first rollout must be verified at DNS, TCP, ACME, TLS, Ingress, and
backend layers separately.
## Rejected alternatives
### TLS termination on Keenetic
Rejected because it makes the router's TLS proxy the certificate owner and was
already observed to return the KeenDNS default `404` instead of reaching the
Traefik host route.
### One public port per domain
Rejected for the standard URL requirement. It would require URLs such as
`https://immich.rudolf.keenetic.link:8443/` and would not serve the default
port `443`.
### DNS-01
Not selected because the requested operating model is HTTP-01. DNS-01 can be
considered later if direct public port `80` cannot be provided.
## Acceptance criteria
- `web` listens on TCP `80` and `websecure` listens on TCP `443`.
- Keenetic forwards TCP `80` and `443` to `192.168.1.114` without TLS
termination.
- The ACME storage is persistent and not committed with credentials.
- `curl -4I http://immich.rudolf.keenetic.link/` returns a permanent redirect to
the equivalent `https://` URL.
- HTTP-01 validation succeeds for every hostname requested in the Ingress.
- `curl --resolve <host>:443:<wan-ip> https://<host>/` completes TLS with a
certificate valid for `<host>` and reaches the expected backend.
- The ACME challenge path remains available on `web` and does not depend on the
KeenDNS web proxy.
+6 -6
View File
@@ -40,17 +40,17 @@ flux get kustomizations -n flux-system
The pinned v2.6.4 release is installed as a separate bootstrap step because
the Flux CRDs must exist before the `GitRepository` and `Kustomization`
objects can be applied. The root Kustomization then creates the source and reconciliation objects for
the K3s `main` branch at the repository root and the connectivity tester
`main` branch under `./k3s`.
objects can be applied. The root Kustomization then creates the source and
reconciliation objects for this repository's `main` branch at the repository
root and the connectivity tester `main` branch under `./k3s`.
## Local bootstrap resources
The cluster configuration is `k3s_rencher/`. Render it before applying:
The cluster configuration is the repository root. Render it before applying:
```sh
kubectl kustomize k3s_rencher
kubectl apply -k k3s_rencher
kubectl kustomize .
kubectl apply -k .
```
The overlay currently installs the read-only agent account and its network
+31
View File
@@ -24,6 +24,24 @@ Run on `rudolf-T`:
```sh
export K3S_TOKEN='<TODO>'
sudo env K3S_TOKEN="$K3S_TOKEN" sh -c '
set -eu
install -d -m 0755 /etc/rancher/k3s
install -d -m 0755 /etc/systemd/system/k3s-agent.service.d
printf "%s\\n" \
"nameserver 192.168.1.1" \
"options timeout:2 attempts:2" \
> /etc/rancher/k3s-resolv.conf
printf "%s\\n" \
"[Service]" \
"BindReadOnlyPaths=/etc/rancher/k3s-resolv.conf:/etc/resolv.conf" \
> /etc/systemd/system/k3s-agent.service.d/10-registry-dns.conf
systemctl daemon-reload
curl -sfL https://get.k3s.io | \
K3S_URL="https://100.70.193.97:6443" \
K3S_TOKEN="$K3S_TOKEN" \
@@ -31,8 +49,11 @@ curl -sfL https://get.k3s.io | \
--node-ip="100.105.201.15" \
--node-external-ip="100.105.201.15" \
--flannel-iface="tailscale0"
'
```
The resolver override is mounted only into the `k3s-agent` service namespace. It does not replace the Ubuntu host resolver for other services. The embedded containerd inherits it for image pulls.
### Join `xu4`
Run on `xu4`:
@@ -70,3 +91,13 @@ kubectl get pods -A -o wide
# Recent warnings/errors across namespaces
kubectl get events -A --sort-by=.lastTimestamp | tail -n 50
# Request a fresh Git fetch. The source update triggers its dependent Kustomization.
kubectl annotate -n flux-system --overwrite gitrepository/k3s reconcile.fluxcd.io/requestedAt="$(date +%s)"
# Compare the latest fetched commit with the last successful application.
kubectl get kustomization k3s -n flux-system -o jsonpath='applied: {.status.lastAppliedRevision}{"\nattempted: "}{.status.lastAttemptedRevision}{"\n"}'
# Show reconciliation conditions and recent errors/events.
kubectl describe kustomization k3s -n flux-system
+132
View File
@@ -0,0 +1,132 @@
# Hermes Swarm to K3s Migration
## Placement and storage
Placement matches the Swarm constraints:
- Hermes and the UI are scheduled on `rudolf-t`.
The Kubernetes deployment uses a static NFS-backed claim:
- `hermes-data-nfs` -> `/root/volumes/hermes/k3s` -> `/opt/data`
The directory is separate from the Swarm Hermes paths:
- `/root/volumes/hermes/gateway`
- `/root/volumes/hermes/agent`
The old `hermes-data` local-path PVC remains declared for rollback and is not
mounted by the Deployment after the NFS switch.
Verify the claims and volume before migration:
```sh
kubectl get pvc -n hermes
kubectl get pv
kubectl get nodes
```
Create the NFS directory on `192.168.1.96` before rollout:
```sh
sudo mkdir -p /root/volumes/hermes/k3s
```
The NFS claim is intentionally separate from the Swarm state. Copy data only
when the Kubernetes instance should inherit a selected subset of the Swarm
state; do not run both instances against the same mutable directory.
Do not run both stacks against the same Telegram bot token or the same
mutable data directory at the same time.
## Copy Docker volumes into PVCs
Run this before applying the Hermes Deployment. First identify the real
Docker volume names; Swarm usually prefixes them with the stack name:
```sh
docker volume ls
docker volume inspect <stack>_hermes_gateway_data
```
Stop the Swarm stack and make sure no K3s Hermes workloads are using the PVCs:
```sh
docker stack services <stack>
docker stack rm <stack>
kubectl -n hermes get pods
```
Create a temporary pod that mounts the claim. The node affinity ensures the
local-path provisioner creates it on `rudolf-t`:
```sh
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: hermes-volume-migration
namespace: hermes
spec:
restartPolicy: Never
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- rudolf-t
containers:
- name: migration
image: alpine:3.20
command: ["sleep", "3600"]
volumeMounts:
- name: hermes-data
mountPath: /target/hermes
volumes:
- name: hermes-data
persistentVolumeClaim:
claimName: hermes-data
EOF
kubectl -n hermes wait --for=condition=Ready pod/hermes-volume-migration \
--timeout=180s
```
From the Docker host, copy the volume as a tar stream. Replace the volume
name with the name reported by `docker volume ls`:
```sh
docker run --rm \
-v '<stack>_hermes_gateway_data:/source:ro' \
alpine:3.20 tar -C /source -cf - . |
kubectl -n hermes exec -i hermes-volume-migration -- \
tar -C /target/hermes -xf -
```
Verify the copied data before removing the temporary pod:
```sh
kubectl -n hermes exec hermes-volume-migration -- du -sh /target/hermes
kubectl -n hermes delete pod hermes-volume-migration
```
Finally deploy Hermes and verify the rollout:
```sh
kubectl apply -k .
kubectl -n hermes rollout status deployment/hermes
kubectl -n hermes get pods,pvc
```
If Docker runs on a different host than `kubectl`, run the `docker run ...`
part on that host and pipe its output over SSH to the machine running
`kubectl`, for example:
```sh
ssh <docker-host> "docker run --rm -v '<stack>_hermes_gateway_data:/source:ro' alpine:3.20 tar -C /source -cf -" |
kubectl -n hermes exec -i hermes-volume-migration -- tar -C /target/hermes -xf -
```
+52
View File
@@ -0,0 +1,52 @@
# Hermes Agent
This deployment runs the Hermes Agent gateway and its Nginx dashboard reverse
proxy in Kubernetes with persistent state.
Hermes is a single gateway process; Telegram, the dashboard, scheduled jobs,
and persisted sessions share the same pod.
## Credentials
Create `hermes-credentials` in the `hermes` namespace with the credentials
appropriate for the selected model provider:
```sh
kubectl -n hermes create secret generic hermes-credentials \
--from-literal=HERMES_DASHBOARD_BASIC_AUTH_USERNAME='<dashboard user>' \
--from-literal=HERMES_DASHBOARD_BASIC_AUTH_PASSWORD='<dashboard password>'
```
Do not commit this Secret; supply it through the cluster's secret-management
workflow.
Hermes egress allows public IPv4 destinations while blocking:
- `192.168.0.0/24`
- `192.168.1.0/24`
- `172.0.0.0/8`
- `100.0.0.0/8`
The policy is defined in [network-policy.yaml](./network-policy.yaml).
## Migration
See [Swarm to K3s migration](./MIGRATION.md) for placement, volume, and
cutover instructions.
## Deploy
Render and inspect the manifests, then let Flux reconcile them:
```sh
kubectl kustomize k3s_rencher/apps/hermes
kubectl apply -k k3s_rencher/apps/hermes
kubectl -n hermes rollout status deployment/hermes
```
The dashboard is exposed through the cluster Traefik Ingress at
`hermes.rudolf.keenetic.link`. The Ingress routes to the `hermes` Service on
port `9119`; Traefik is allowed to reach that port by the namespace NetworkPolicy.
The router must keep the public web application entry pointed at
`xu4_lan` (`192.168.1.114`) on HTTP port `80`. Do not point the router at the
Kubernetes Service port `9119` or at a Tailscale node address.
+77
View File
@@ -0,0 +1,77 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hermes
labels:
app.kubernetes.io/name: hermes
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: hermes
template:
metadata:
labels:
app.kubernetes.io/name: hermes
spec:
automountServiceAccountToken: false
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- rudolf-t
securityContext:
runAsNonRoot: false
runAsUser: 0
runAsGroup: 0
containers:
- name: hermes
image: nousresearch/hermes-agent:v2026.8.3
imagePullPolicy: IfNotPresent
args:
- gateway
- run
env:
- name: HERMES_DASHBOARD
value: "1"
- name: HERMES_DASHBOARD_HOST
value: 0.0.0.0
- name: HERMES_YOLO_MODE
value: "1"
- name: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
valueFrom:
secretKeyRef:
name: hermes-credentials
key: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
- name: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
valueFrom:
secretKeyRef:
name: hermes-credentials
key: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
ports:
- name: dashboard
containerPort: 9119
protocol: TCP
volumeMounts:
- name: data
mountPath: /opt/data
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
add:
- CHOWN
- DAC_OVERRIDE
- SETGID
- SETUID
volumes:
- name: data
persistentVolumeClaim:
claimName: hermes-data-nfs
@@ -0,0 +1,97 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: brencher-state-proxy-config
data:
default.conf: |
server {
listen 8080;
server_name _;
location = /healthz {
access_log off;
return 200 "ok\n";
add_header Content-Type text/plain;
}
location = /state {
proxy_pass https://brencher.rudolf.keenetic.link/state;
proxy_http_version 1.1;
proxy_ssl_server_name on;
proxy_ssl_name brencher.rudolf.keenetic.link;
proxy_set_header Host brencher.rudolf.keenetic.link;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_connect_timeout 5s;
proxy_read_timeout 30s;
proxy_send_timeout 10s;
proxy_buffering off;
add_header Cache-Control "no-store" always;
}
location = /state2 {
proxy_pass http://192.168.1.96:5002/state;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_connect_timeout 5s;
proxy_read_timeout 30s;
proxy_send_timeout 10s;
proxy_buffering off;
add_header Cache-Control "no-store" always;
}
location / {
access_log off;
return 403 "forbidden: only /state and /state2 are allowed through this proxy\n";
add_header Content-Type text/plain;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: brencher-state-proxy
labels:
app.kubernetes.io/name: brencher-state-proxy
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: brencher-state-proxy
template:
metadata:
labels:
app.kubernetes.io/name: brencher-state-proxy
spec:
automountServiceAccountToken: false
containers:
- name: nginx
image: nginx:alpine
ports:
- name: http
containerPort: 8080
protocol: TCP
volumeMounts:
- name: config
mountPath: /etc/nginx/conf.d/default.conf
subPath: default.conf
readOnly: true
volumes:
- name: config
configMap:
name: brencher-state-proxy-config
---
apiVersion: v1
kind: Service
metadata:
name: brencher-state-proxy
labels:
app.kubernetes.io/name: brencher-state-proxy
spec:
selector:
app.kubernetes.io/name: brencher-state-proxy
ports:
- name: http
port: 8080
targetPort: http
protocol: TCP
@@ -0,0 +1,24 @@
apiVersion: v1
kind: Service
metadata:
name: docker-socket-proxy
labels:
app.kubernetes.io/name: docker-socket-proxy
spec:
ports:
- name: docker
port: 2375
targetPort: 2375
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: docker-socket-proxy
subsets:
- addresses:
- ip: 192.168.1.96
ports:
- name: docker
port: 2375
protocol: TCP
@@ -0,0 +1,11 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hermes-gateway
spec:
template:
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: hermes-data-nfs
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: hermes-gateway
labels:
app.kubernetes.io/name: hermes-gateway
spec:
selector:
app.kubernetes.io/name: hermes-gateway
ports:
- name: dashboard
port: 9119
targetPort: dashboard
protocol: TCP
@@ -0,0 +1,91 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hermes-gateway
labels:
app.kubernetes.io/name: hermes-gateway
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: hermes-gateway
template:
metadata:
labels:
app.kubernetes.io/name: hermes-gateway
spec:
automountServiceAccountToken: false
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- rudolf-t
securityContext:
runAsNonRoot: false
runAsUser: 0
runAsGroup: 0
containers:
- name: hermes-gateway
image: nousresearch/hermes-agent:v2026.8.3
imagePullPolicy: IfNotPresent
args:
- gateway
- run
env:
- name: HERMES_DASHBOARD
value: "1"
- name: HERMES_DASHBOARD_HOST
value: 0.0.0.0
- name: HERMES_YOLO_MODE
value: "1"
- name: HERMES_AGENT_URL
value: http://hermes-agent:8000
- name: HTTP_PROXY
value: http://squid:3128
- name: HTTPS_PROXY
value: http://squid:3128
- name: http_proxy
value: http://squid:3128
- name: https_proxy
value: http://squid:3128
- name: NO_PROXY
value: localhost,127.0.0.1,.local,squid,hermes-gateway,hermes-agent,hermes-ui,brencher-state-proxy,docker-socket-proxy,10.0.0.0/8
- name: no_proxy
value: localhost,127.0.0.1,.local,squid,hermes-gateway,hermes-agent,hermes-ui,brencher-state-proxy,docker-socket-proxy,10.0.0.0/8
- name: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
valueFrom:
secretKeyRef:
name: hermes-credentials
key: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
- name: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
valueFrom:
secretKeyRef:
name: hermes-credentials
key: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
ports:
- name: dashboard
containerPort: 9119
protocol: TCP
volumeMounts:
- name: data
mountPath: /opt/data
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
add:
- CHOWN
- DAC_OVERRIDE
- SETGID
- SETUID
volumes:
- name: data
persistentVolumeClaim:
claimName: hermes-data
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gateway.yaml
- gateway-service.yaml
- ui.yaml
- squid.yaml
- docker-socket-proxy.yaml
- brencher-state-proxy.yaml
- network-policy.yaml
patches:
- path: gateway-pvc-patch.yaml
@@ -0,0 +1,124 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hermes-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: hermes-gateway
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 192.168.1.1/32
ports:
- protocol: TCP
port: 443
- to:
- ipBlock:
cidr: 100.70.193.97/32
ports:
- protocol: TCP
port: 6443
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 192.168.0.0/24
- 192.168.1.0/24
- 172.0.0.0/8
- 100.0.0.0/8
- to:
- ipBlock:
cidr: 192.168.1.96/32
ports:
- protocol: TCP
port: 2375
- to:
- ipBlock:
cidr: 100.105.201.15/32
ports:
- protocol: TCP
port: 9875
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hermes-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: hermes-gateway
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/instance: traefik-kube-system
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 9119
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: hermes-ui
ports:
- protocol: TCP
port: 9119
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: brencher-state-proxy-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: brencher-state-proxy
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 192.168.0.0/24
- 192.168.1.0/24
ports:
- protocol: TCP
port: 443
- to:
- ipBlock:
cidr: 192.168.1.96/32
ports:
- protocol: TCP
port: 5002
@@ -0,0 +1,80 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: squid-config
data:
squid.conf: |
acl localnet src 10.0.0.0/8
acl localnet src 172.16.0.0/12
acl localnet src 192.168.0.0/16
acl allowed_domains dstdomain .minimax.io .minimax.chat .minimax.com .githubusercontent.com deb.debian.org gitea.rudolf.keenetic.link
acl telegram_domains dstdomain .telegram.org .t.me
acl stt_pkg_domains dstdomain .pypi.org .files.pythonhosted.org .huggingface.co .github.com .models.dev .registry.npmjs.org .cdn-lfs.huggingface.co .hf.co .xethub.hf.co .cas-server.xethub.hf.co
acl copilot_domains dstdomain api.github.com copilot-proxy.githubusercontent.com .githubcopilot.com copilot-telemetry.githubusercontent.com default.exp-tas.com marketplace.visualstudio.com vscode.blob.core.windows.net .githubassets.com .vscode-unpkg.net
acl telegram_ips dst 91.108.4.0/22 91.108.8.0/22 91.108.12.0/22 91.108.16.0/22 91.108.20.0/22 91.108.56.0/22 149.154.160.0/20 185.76.151.0/24
acl CONNECT method CONNECT
acl SSL_ports port 443
acl Safe_ports port 80 443
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet allowed_domains
http_access allow localnet telegram_domains
http_access allow localnet stt_pkg_domains
http_access allow localnet copilot_domains
http_access allow localnet telegram_ips
http_access deny all
http_port 3128
access_log stdio:/var/log/squid/access.log
cache deny all
cache_mem 0 MB
cache_dir null /tmp
coredump_dir /var/spool/squid
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: squid
labels:
app.kubernetes.io/name: squid
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: squid
template:
metadata:
labels:
app.kubernetes.io/name: squid
spec:
automountServiceAccountToken: false
containers:
- name: squid
image: ubuntu/squid:latest
ports:
- name: proxy
containerPort: 3128
protocol: TCP
volumeMounts:
- name: config
mountPath: /etc/squid/squid.conf
subPath: squid.conf
readOnly: true
volumes:
- name: config
configMap:
name: squid-config
---
apiVersion: v1
kind: Service
metadata:
name: squid
labels:
app.kubernetes.io/name: squid
spec:
selector:
app.kubernetes.io/name: squid
ports:
- name: proxy
port: 3128
targetPort: proxy
protocol: TCP
@@ -0,0 +1,19 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hermes-ui
labels:
app.kubernetes.io/name: hermes
spec:
ingressClassName: traefik
rules:
- host: hermes.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hermes
port:
name: dashboard
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: hermes-generic1
resources:
- ../generic-base
- namespace.yaml
- nfs-pv.yaml
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: hermes-generic1
@@ -0,0 +1,29 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: hermes-generic1-data-nfs
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: hermes-generic1-nfs
mountOptions:
- nfsvers=3
nfs:
server: 192.168.1.96
path: /root/volumes/hermes/generic1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hermes-data-nfs
spec:
accessModes:
- ReadWriteOnce
storageClassName: hermes-generic1-nfs
volumeName: hermes-generic1-data-nfs
resources:
requests:
storage: 10Gi
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: hermes-generic2
resources:
- ../generic-base
- namespace.yaml
- nfs-pv.yaml
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: hermes-generic2
@@ -0,0 +1,29 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: hermes-generic2-data-nfs
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: hermes-generic2-nfs
mountOptions:
- nfsvers=3
nfs:
server: 192.168.1.96
path: /root/volumes/hermes/generic2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hermes-data-nfs
spec:
accessModes:
- ReadWriteOnce
storageClassName: hermes-generic2-nfs
volumeName: hermes-generic2-data-nfs
resources:
requests:
storage: 10Gi
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: hermes
resources:
- namespace.yaml
- pvc.yaml
- nfs-pv.yaml
- nfs-pvc.yaml
- ui.yaml
- deployment.yaml
- service.yaml
- network-policy.yaml
+4
View File
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: hermes
@@ -0,0 +1,88 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hermes-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: hermes
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 192.168.1.1/32
ports:
- protocol: TCP
port: 443
- to:
- ipBlock:
cidr: 100.70.193.97/32
ports:
- protocol: TCP
port: 6443
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 192.168.0.0/24
- 192.168.1.0/24
- 172.0.0.0/8
- 100.0.0.0/8
- to:
- ipBlock:
cidr: 192.168.1.96/32
ports:
- protocol: TCP
port: 2375
- to:
- ipBlock:
cidr: 100.105.201.15/32
ports:
- protocol: TCP
port: 9875
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hermes-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: hermes
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: hermes
ports:
- protocol: TCP
port: 9119
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/instance: traefik-kube-system
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 9119
+16
View File
@@ -0,0 +1,16 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: hermes-data-nfs
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: hermes-nfs
mountOptions:
- nfsvers=3
nfs:
server: 192.168.1.96
path: /root/volumes/hermes/k3s
+12
View File
@@ -0,0 +1,12 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hermes-data-nfs
spec:
accessModes:
- ReadWriteOnce
storageClassName: hermes-nfs
volumeName: hermes-data-nfs
resources:
requests:
storage: 10Gi
+10
View File
@@ -0,0 +1,10 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hermes-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: hermes
labels:
app.kubernetes.io/name: hermes
spec:
selector:
app.kubernetes.io/name: hermes
ports:
- name: dashboard
port: 9119
targetPort: dashboard
protocol: TCP
+26
View File
@@ -0,0 +1,26 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hermes-ui
labels:
app.kubernetes.io/name: hermes
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
spec:
ingressClassName: traefik
tls:
- hosts:
- hermes.rudolf.keenetic.link
rules:
- host: hermes.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hermes
port:
name: dashboard
+1 -1
View File
@@ -7,4 +7,4 @@ spec:
interval: 1m
ref:
branch: main
url: https://gitea.rudolf.keenetic.link/org1/k3s.git
url: https://gitea.rudolf.keenetic.link/org1/infra.git
@@ -5,14 +5,108 @@ metadata:
namespace: agent-gateway
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: agent-readonly-all-except-secrets
rules:
- apiGroups:
- ""
resources:
- bindings
- componentstatuses
- configmaps
- endpoints
- events
- limitranges
- namespaces
- namespaces/status
- nodes
- nodes/status
- persistentvolumeclaims
- persistentvolumeclaims/status
- persistentvolumes
- persistentvolumes/status
- pods
- pods/ephemeralcontainers
- pods/log
- pods/status
- podtemplates
- replicationcontrollers
- replicationcontrollers/status
- resourcequotas
- resourcequotas/status
- serviceaccounts
- services
- services/status
verbs:
- get
- list
- watch
- apiGroups:
- apiregistration.k8s.io
- apps
- admissionregistration.k8s.io
- apiextensions.k8s.io
- authentication.k8s.io
- authorization.k8s.io
- autoscaling
- batch
- certificates.k8s.io
- coordination.k8s.io
- discovery.k8s.io
- events.k8s.io
- flowcontrol.apiserver.k8s.io
- gateway.networking.k8s.io
- helm.cattle.io
- helm.toolkit.fluxcd.io
- hub.traefik.io
- image.toolkit.fluxcd.io
- k3s.cattle.io
- kustomize.toolkit.fluxcd.io
- metrics.k8s.io
- networking.k8s.io
- node.k8s.io
- notification.toolkit.fluxcd.io
- policy
- rbac.authorization.k8s.io
- resource.k8s.io
- scheduling.k8s.io
- source.toolkit.fluxcd.io
- storage.k8s.io
- traefik.io
resources:
- "*"
verbs:
- get
- list
- watch
- nonResourceURLs:
- /api
- /api/*
- /apis
- /apis/*
- /healthz
- /healthz/*
- /livez
- /livez/*
- /openapi
- /openapi/*
- /readyz
- /readyz/*
- /version
- /version/*
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: agent-readonly-view
name: agent-readonly-all-except-secrets
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: view
name: agent-readonly-all-except-secrets
subjects:
- kind: ServiceAccount
name: agent-readonly
namespace: agent-gateway
namespace: agent-gateway
@@ -0,0 +1,128 @@
# Shared public ingress
The common Traefik entrypoint routes public domains to Kubernetes and Swarm
backends.
| Domain | Backend |
| --- | --- |
| `hermes.rudolf.keenetic.link` | Kubernetes Service `hermes:9119` |
| `brencher.rudolf.keenetic.link` | Swarm `192.168.1.96:5001` |
| `immich.rudolf.keenetic.link` | Swarm `192.168.1.96:2283` |
| `gitea.rudolf.keenetic.link` | Swarm `192.168.1.96:3002` |
The Swarm backends are represented by selectorless Kubernetes Services and
static Endpoints. The Swarm ports must remain reachable from the Traefik node
at `192.168.1.96`.
## HTTPS target configuration
The target design is TCP forwarding to Traefik, with TLS termination in
Traefik:
```text
WAN TCP 80 -> 192.168.1.114:80
WAN TCP 443 -> 192.168.1.114:443
```
Configure the Keenetic entries as raw TCP forwarding. Do not use the KeenDNS
HTTPS proxy, terminate TLS on the router, rewrite SNI, or replace the HTTP
`Host` header.
Traefik must provide:
- `web` on TCP `80` for HTTP and ACME HTTP-01;
- `websecure` on TCP `443` for TLS termination;
- a persistent ACME storage file, for example `/data/acme.json`;
- an ACME resolver with `httpChallenge.entryPoint: web`;
- a non-placeholder registration email supplied outside Git.
The exact `HelmChartConfig` values must match the installed K3s Traefik chart
version. Inspect the live `HelmChart/traefik` values before applying a chart
override; do not assume that a values schema from another K3s release is
compatible.
For each public Ingress, pin the TLS route to `websecure`. The separate
`http-redirect.yaml` resource attaches a `redirectScheme` middleware to
`web`-only routes. Keeping the HTTP redirect separate prevents the internal
ACME challenge router from being sent to a backend.
The resulting TLS Ingress shape is:
```yaml
metadata:
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
spec:
tls:
- hosts:
- immich.rudolf.keenetic.link
```
With a Traefik ACME resolver, the certificate is stored in the configured
persistent ACME storage rather than committed to Git. If the installed chart
requires a Kubernetes Secret for a different TLS mode, provision that Secret
through the approved secret-management path. Do not commit `acme.json`, a
private key, an API token, or an unencrypted Kubernetes Secret.
## Declarative traffic configuration
The repository configuration is in `traefik-https-config.yaml`. It extends the
installed K3s `HelmChart/traefik` with:
- `web` exposed on TCP `80` for the HTTP redirect and ACME HTTP-01;
- `websecure` exposed on TCP `443` for TLS termination;
- a permanent HTTP-to-HTTPS `redirectScheme` middleware on web-only routes;
- the `letsencrypt` ACME resolver using HTTP-01 through `web` (Traefik's
ACME challenge handling remains available on the `web` entrypoint);
- persistent `/data/acme.json` storage through a `1Gi` PVC.
The resolver uses the operator-supplied registration email from the
configuration. The email is not a credential; the ACME account state and
certificate private keys remain in persistent `/data/acme.json` storage and
must not be committed to Git. The TLS annotations and host lists are present
in both the shared public Ingress and Hermes-specific Ingress.
## Rollout procedure
1. Verify that every hostname resolves to the router's current public address.
2. Configure Keenetic WAN TCP `80` and `443` forwarding to
`192.168.1.114:80` and `192.168.1.114:443`.
3. Confirm from an external network that TCP `80` reaches Traefik. The
KeenDNS default `404` is not proof of correct forwarding.
4. Configure the Traefik `websecure` entrypoint, persistent ACME storage, and
HTTP-01 resolver through the repository's GitOps path.
5. Add TLS annotations and host entries to the public Ingress resources.
6. Apply through Flux only after rendering and reviewing the complete diff.
7. Check the Traefik logs and ACME resource/state for the challenge result.
8. Test HTTP-01 reachability:
```sh
curl -4v http://immich.rudolf.keenetic.link/.well-known/acme-challenge/test
```
This request must reach Traefik, not the KeenDNS web proxy.
9. Test TLS and SNI directly against the WAN address:
```sh
curl -4vk \
--resolve immich.rudolf.keenetic.link:443:<wan-ip> \
https://immich.rudolf.keenetic.link/
```
10. Verify the certificate SAN, the Traefik route, the HTTP status, and the
Immich backend separately. Repeat the TLS/SNI test for every public
hostname.
## Failure boundaries
- DNS resolves incorrectly: fix DNS before changing Kubernetes.
- TCP `80` or `443` is refused: fix Keenetic forwarding or WAN reachability.
- KeenDNS certificate/default `404` appears: traffic still uses the router
proxy instead of TCP forwarding.
- ACME challenge fails: inspect port `80`, Host preservation, and Traefik
resolver/storage before changing backend routes.
- TLS succeeds but the route is `404`/`502`: inspect the Ingress, Service,
Endpoints, and Traefik access log separately.
@@ -0,0 +1,63 @@
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: http-to-https
namespace: hermes
spec:
redirectScheme:
scheme: https
permanent: true
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: http-redirect
namespace: hermes
labels:
app.kubernetes.io/name: http-redirect
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
traefik.ingress.kubernetes.io/router.middlewares: hermes-http-to-https@kubernetescrd
spec:
ingressClassName: traefik
rules:
- host: hermes.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hermes
port:
name: dashboard
- host: brencher.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: brencher-swarm
port:
name: http
- host: immich.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: immich-swarm
port:
name: http
- host: gitea.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: gitea-swarm
port:
name: http
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- swarm-backends.yaml
- traefik-https-config.yaml
- http-redirect.yaml
- public-ingress.yaml
@@ -0,0 +1,49 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: public-ingress
namespace: hermes
labels:
app.kubernetes.io/name: public-ingress
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
spec:
ingressClassName: traefik
tls:
- hosts:
- brencher.rudolf.keenetic.link
- immich.rudolf.keenetic.link
- gitea.rudolf.keenetic.link
rules:
- host: brencher.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: brencher-swarm
port:
name: http
- host: immich.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: immich-swarm
port:
name: http
- host: gitea.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: gitea-swarm
port:
name: http
@@ -0,0 +1,80 @@
apiVersion: v1
kind: Service
metadata:
namespace: hermes
name: brencher-swarm
labels:
app.kubernetes.io/name: brencher-swarm
spec:
ports:
- name: http
port: 5001
targetPort: 5001
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
namespace: hermes
name: brencher-swarm
subsets:
- addresses:
- ip: 192.168.1.96
ports:
- name: http
port: 5001
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
namespace: hermes
name: immich-swarm
labels:
app.kubernetes.io/name: immich-swarm
spec:
ports:
- name: http
port: 2283
targetPort: 2283
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
namespace: hermes
name: immich-swarm
subsets:
- addresses:
- ip: 192.168.1.96
ports:
- name: http
port: 2283
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
namespace: hermes
name: gitea-swarm
labels:
app.kubernetes.io/name: gitea-swarm
spec:
ports:
- name: http
port: 3002
targetPort: 3002
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
namespace: hermes
name: gitea-swarm
subsets:
- addresses:
- ip: 192.168.1.96
ports:
- name: http
port: 3002
protocol: TCP
@@ -0,0 +1,36 @@
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
nodeSelector:
kubernetes.io/hostname: xu4
deployment:
replicas: 1
ports:
web:
port: 8000
exposedPort: 80
expose:
default: true
protocol: TCP
websecure:
port: 8443
exposedPort: 443
expose:
default: true
protocol: TCP
additionalArguments:
- "--entrypoints.web.address=:8000/tcp"
- "--entrypoints.websecure.address=:8443/tcp"
- "--certificatesresolvers.letsencrypt.acme.email=rudolfss13@gmail.com"
- "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
- "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
persistence:
enabled: true
name: data
path: /data
accessMode: ReadWriteOnce
size: 1Gi
+4
View File
@@ -4,3 +4,7 @@ resources:
- k3s_rencher/flux-system
- k3s_rencher/infrastructure/dns
- k3s_rencher/infrastructure/agent-gateway
- k3s_rencher/infrastructure/ingress
- k3s_rencher/apps/hermes
- k3s_rencher/apps/hermes/generic1
- k3s_rencher/apps/hermes/generic2