[ready-to-test] refactor(k3s): base Hermes infra and add generic overlay (#17)

Replace the duplicated hermes/hermes_generic manifests with a shared Kubernetes base derived from legacy hermes_infra. Add hermes_infra and generic overlays with separate namespaces, NFS PV/PVC state, and credentials. Preserve the external Swarm docker socket through a namespaced Service/Endpoints because K3s uses containerd and has no local Docker socket. Sarg is intentionally deferred until its dynamically tagged image has a stable Kubernetes reference.

---------

Co-authored-by: hermes <hermes@rudolf.keenetic.link>
Reviewed-on: #17
Reviewed-by: admin <rudolfss13@gmail.com>
Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
This commit was merged in pull request #17.
This commit is contained in:
2026-11-21 13:20:45 +01:00
committed by admin
co-authored by hermes
parent 5e314e17d0
commit 95edba82bf
17 changed files with 567 additions and 2 deletions
+13 -2
View File
@@ -37,7 +37,13 @@ This file records repository decisions and the guidance that follows from them.
- **Decision:** Configuration should be applied only through Git pushes. Do not apply configuration directly to infrastructure, even when access is available. An agent may apply configs directly tom infrastructure only when explicitly requested.
### DEC-009: Manage web application domains through Keenetic Hero 4G
### DEC-009: Use separate NFS state for Kubernetes Hermes
- **Context:** Kubernetes Hermes must not share mutable state with the legacy Swarm Hermes instance. K3s `local-path` storage is node-local and does not provide the desired storage layout.
- **Decision:** Kubernetes Hermes uses a static NFS-backed `hermes-data-nfs` claim from `192.168.1.96:/root/volumes/hermes/k3s`, mounted at `/opt/data`. The volume uses NFSv3 and `Retain` reclaim policy. The Swarm directories `/root/volumes/hermes/gateway` and `/root/volumes/hermes/agent` remain separate.
- **Consequences:** The NFS export must allow the current Kubernetes node addresses `100.70.193.97`, `100.105.201.15`, and `100.96.92.102`, in addition to the existing Swarm LAN allowance `192.168.1.0/24`. Kubernetes and Swarm Hermes must not use the same mutable state directory concurrently.
### DEC-010: Manage web application domains through Keenetic Hero 4G
- **Context:** Remote access to web applications in the network is managed through the Keenetic Hero 4G router's web application access settings.
- **Decision:** The router-managed web application mappings are:
@@ -53,7 +59,12 @@ This file records repository decisions and the guidance that follows from them.
## Draft
No draft decisions recorded.
### DEC-011: Add authenticated NFS access for Hermes
- **Context:** The current NFS access model relies on export client allowlists and Unix filesystem permissions. It does not provide NFS username/password authentication.
- **Decision:** Defer NFS authentication. Do not add Kerberos or store NFS credentials in Kubernetes as part of the current Hermes storage migration.
- **Open options:** Narrow exports to exact client addresses, replace `no_root_squash` with `root_squash` after validating Hermes UID/GID behavior, or evaluate Kerberos-backed NFS with `sec=krb5`/`sec=krb5p` in a separate change.
- **Status:** Draft; the current IP-restricted export model remains in use.
## Declined
@@ -0,0 +1,97 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: brencher-state-proxy-config
data:
default.conf: |
server {
listen 8080;
server_name _;
location = /healthz {
access_log off;
return 200 "ok\n";
add_header Content-Type text/plain;
}
location = /state {
proxy_pass https://brencher.rudolf.keenetic.link/state;
proxy_http_version 1.1;
proxy_ssl_server_name on;
proxy_ssl_name brencher.rudolf.keenetic.link;
proxy_set_header Host brencher.rudolf.keenetic.link;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_connect_timeout 5s;
proxy_read_timeout 30s;
proxy_send_timeout 10s;
proxy_buffering off;
add_header Cache-Control "no-store" always;
}
location = /state2 {
proxy_pass http://192.168.1.96:5002/state;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_connect_timeout 5s;
proxy_read_timeout 30s;
proxy_send_timeout 10s;
proxy_buffering off;
add_header Cache-Control "no-store" always;
}
location / {
access_log off;
return 403 "forbidden: only /state and /state2 are allowed through this proxy\n";
add_header Content-Type text/plain;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: brencher-state-proxy
labels:
app.kubernetes.io/name: brencher-state-proxy
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: brencher-state-proxy
template:
metadata:
labels:
app.kubernetes.io/name: brencher-state-proxy
spec:
automountServiceAccountToken: false
containers:
- name: nginx
image: nginx:alpine
ports:
- name: http
containerPort: 8080
protocol: TCP
volumeMounts:
- name: config
mountPath: /etc/nginx/conf.d/default.conf
subPath: default.conf
readOnly: true
volumes:
- name: config
configMap:
name: brencher-state-proxy-config
---
apiVersion: v1
kind: Service
metadata:
name: brencher-state-proxy
labels:
app.kubernetes.io/name: brencher-state-proxy
spec:
selector:
app.kubernetes.io/name: brencher-state-proxy
ports:
- name: http
port: 8080
targetPort: http
protocol: TCP
@@ -0,0 +1,24 @@
apiVersion: v1
kind: Service
metadata:
name: docker-socket-proxy
labels:
app.kubernetes.io/name: docker-socket-proxy
spec:
ports:
- name: docker
port: 2375
targetPort: 2375
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: docker-socket-proxy
subsets:
- addresses:
- ip: 192.168.1.96
ports:
- name: docker
port: 2375
protocol: TCP
@@ -0,0 +1,11 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hermes-gateway
spec:
template:
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: hermes-data-nfs
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: hermes-gateway
labels:
app.kubernetes.io/name: hermes-gateway
spec:
selector:
app.kubernetes.io/name: hermes-gateway
ports:
- name: dashboard
port: 9119
targetPort: dashboard
protocol: TCP
@@ -0,0 +1,91 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hermes-gateway
labels:
app.kubernetes.io/name: hermes-gateway
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: hermes-gateway
template:
metadata:
labels:
app.kubernetes.io/name: hermes-gateway
spec:
automountServiceAccountToken: false
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- rudolf-t
securityContext:
runAsNonRoot: false
runAsUser: 0
runAsGroup: 0
containers:
- name: hermes-gateway
image: nousresearch/hermes-agent:v2026.8.3
imagePullPolicy: IfNotPresent
args:
- gateway
- run
env:
- name: HERMES_DASHBOARD
value: "1"
- name: HERMES_DASHBOARD_HOST
value: 0.0.0.0
- name: HERMES_YOLO_MODE
value: "1"
- name: HERMES_AGENT_URL
value: http://hermes-agent:8000
- name: HTTP_PROXY
value: http://squid:3128
- name: HTTPS_PROXY
value: http://squid:3128
- name: http_proxy
value: http://squid:3128
- name: https_proxy
value: http://squid:3128
- name: NO_PROXY
value: localhost,127.0.0.1,.local,squid,hermes-gateway,hermes-agent,hermes-ui,brencher-state-proxy,docker-socket-proxy,10.0.0.0/8
- name: no_proxy
value: localhost,127.0.0.1,.local,squid,hermes-gateway,hermes-agent,hermes-ui,brencher-state-proxy,docker-socket-proxy,10.0.0.0/8
- name: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
valueFrom:
secretKeyRef:
name: hermes-credentials
key: HERMES_DASHBOARD_BASIC_AUTH_USERNAME
- name: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
valueFrom:
secretKeyRef:
name: hermes-credentials
key: HERMES_DASHBOARD_BASIC_AUTH_PASSWORD
ports:
- name: dashboard
containerPort: 9119
protocol: TCP
volumeMounts:
- name: data
mountPath: /opt/data
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
add:
- CHOWN
- DAC_OVERRIDE
- SETGID
- SETUID
volumes:
- name: data
persistentVolumeClaim:
claimName: hermes-data
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gateway.yaml
- gateway-service.yaml
- ui.yaml
- squid.yaml
- docker-socket-proxy.yaml
- brencher-state-proxy.yaml
- network-policy.yaml
patches:
- path: gateway-pvc-patch.yaml
@@ -0,0 +1,124 @@
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hermes-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: hermes-gateway
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 192.168.1.1/32
ports:
- protocol: TCP
port: 443
- to:
- ipBlock:
cidr: 100.70.193.97/32
ports:
- protocol: TCP
port: 6443
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 192.168.0.0/24
- 192.168.1.0/24
- 172.0.0.0/8
- 100.0.0.0/8
- to:
- ipBlock:
cidr: 192.168.1.96/32
ports:
- protocol: TCP
port: 2375
- to:
- ipBlock:
cidr: 100.105.201.15/32
ports:
- protocol: TCP
port: 9875
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: hermes-ingress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: hermes-gateway
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/instance: traefik-kube-system
app.kubernetes.io/name: traefik
ports:
- protocol: TCP
port: 9119
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: hermes-ui
ports:
- protocol: TCP
port: 9119
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: brencher-state-proxy-egress
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: brencher-state-proxy
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 192.168.0.0/24
- 192.168.1.0/24
ports:
- protocol: TCP
port: 443
- to:
- ipBlock:
cidr: 192.168.1.96/32
ports:
- protocol: TCP
port: 5002
@@ -0,0 +1,80 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: squid-config
data:
squid.conf: |
acl localnet src 10.0.0.0/8
acl localnet src 172.16.0.0/12
acl localnet src 192.168.0.0/16
acl allowed_domains dstdomain .minimax.io .minimax.chat .minimax.com .githubusercontent.com deb.debian.org gitea.rudolf.keenetic.link
acl telegram_domains dstdomain .telegram.org .t.me
acl stt_pkg_domains dstdomain .pypi.org .files.pythonhosted.org .huggingface.co .github.com .models.dev .registry.npmjs.org .cdn-lfs.huggingface.co .hf.co .xethub.hf.co .cas-server.xethub.hf.co
acl copilot_domains dstdomain api.github.com copilot-proxy.githubusercontent.com .githubcopilot.com copilot-telemetry.githubusercontent.com default.exp-tas.com marketplace.visualstudio.com vscode.blob.core.windows.net .githubassets.com .vscode-unpkg.net
acl telegram_ips dst 91.108.4.0/22 91.108.8.0/22 91.108.12.0/22 91.108.16.0/22 91.108.20.0/22 91.108.56.0/22 149.154.160.0/20 185.76.151.0/24
acl CONNECT method CONNECT
acl SSL_ports port 443
acl Safe_ports port 80 443
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet allowed_domains
http_access allow localnet telegram_domains
http_access allow localnet stt_pkg_domains
http_access allow localnet copilot_domains
http_access allow localnet telegram_ips
http_access deny all
http_port 3128
access_log stdio:/var/log/squid/access.log
cache deny all
cache_mem 0 MB
cache_dir null /tmp
coredump_dir /var/spool/squid
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: squid
labels:
app.kubernetes.io/name: squid
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: squid
template:
metadata:
labels:
app.kubernetes.io/name: squid
spec:
automountServiceAccountToken: false
containers:
- name: squid
image: ubuntu/squid:latest
ports:
- name: proxy
containerPort: 3128
protocol: TCP
volumeMounts:
- name: config
mountPath: /etc/squid/squid.conf
subPath: squid.conf
readOnly: true
volumes:
- name: config
configMap:
name: squid-config
---
apiVersion: v1
kind: Service
metadata:
name: squid
labels:
app.kubernetes.io/name: squid
spec:
selector:
app.kubernetes.io/name: squid
ports:
- name: proxy
port: 3128
targetPort: proxy
protocol: TCP
@@ -0,0 +1,19 @@
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hermes-ui
labels:
app.kubernetes.io/name: hermes
spec:
ingressClassName: traefik
rules:
- host: hermes.rudolf.keenetic.link
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: hermes
port:
name: dashboard
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: hermes_generic1
resources:
- ../generic-base
- namespace.yaml
- nfs-pv.yaml
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: hermes_generic1
@@ -0,0 +1,29 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: hermes-generic1-data-nfs
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: hermes-generic1-nfs
mountOptions:
- nfsvers=3
nfs:
server: 192.168.1.96
path: /root/volumes/hermes/generic1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hermes-data-nfs
spec:
accessModes:
- ReadWriteOnce
storageClassName: hermes-generic1-nfs
volumeName: hermes-generic1-data-nfs
resources:
requests:
storage: 10Gi
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: hermes_generic2
resources:
- ../generic-base
- namespace.yaml
- nfs-pv.yaml
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: hermes_generic2
@@ -0,0 +1,29 @@
apiVersion: v1
kind: PersistentVolume
metadata:
name: hermes-generic2-data-nfs
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: hermes-generic2-nfs
mountOptions:
- nfsvers=3
nfs:
server: 192.168.1.96
path: /root/volumes/hermes/generic2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hermes-data-nfs
spec:
accessModes:
- ReadWriteOnce
storageClassName: hermes-generic2-nfs
volumeName: hermes-generic2-data-nfs
resources:
requests:
storage: 10Gi
+2
View File
@@ -5,3 +5,5 @@ resources:
- k3s_rencher/infrastructure/dns
- k3s_rencher/infrastructure/agent-gateway
- k3s_rencher/apps/hermes
- k3s_rencher/apps/hermes/generic1
- k3s_rencher/apps/hermes/generic2