hermes_squid: open docker_socket_proxy to full read-only scope #1
Closed
hermes
wants to merge 9 commits from
feat/docker-socket-proxy-readonly into master
pull from: feat/docker-socket-proxy-readonly
merge into: :master
:master
:feat/gitea-second-runner-rudolf-t
:feat/hermes_squid-kubernetes-api-proxy
:fix/runner-rudolf-t-internal-gitea
:chore/move-connectivity-tester
:auto-3041db43-d17a9362-e9de263e
:feat/hermes-rudolf-t
:auto-d17a9362
:auto-6511ae28
:auto-98e6f251
:auto-82487b15
:auto-e3f195f4
:auto-7ff2762f
:auto-2ec04bb6
:feat/connectivity-tester-release-workflow-admin
:auto-558e1368
:auto-be0b9518
:auto-525201f6
:auto-418ecdd8
:auto-418ecdd8-9436806f
:auto-8ecc6f04
:auto-9436806f
:auto-test-74229c1
:auto-703e0198
:release/0.1.0
:feat/monitoring-prod-watchdog
:fix/gitea-act-runner-cache-server
:docs/hermes_infra-gateway-lag-recovery
:docs/hermes_infra-telegram-mapping
:fix/hermes-gateway-memory-pids
:fix/hermes-gateway-healthcheck
:feat/docker-socket-proxy-readonly
:docs/architecture-diagram-cleanup
:docs/architecture-diagram
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
12bacebdd9 |
hermes_squid: open docker_socket_proxy to full read-only scope
Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1 so hermes-agent can inspect running stacks (e.g. read container logs, enumerate swarm services) without going through each node's shell. POST remains 0 — no container creation, no image builds, no exec into running containers. Anything that mutates state stays blocked at the proxy level. Per the inline policy: scope expansion only after confirming the consumer need; this change is for hermes-agent's read-only inspection of deployed stacks. |
||
|
|
2784be9dc7 |
fix(hermes_squid): harden NO_PROXY for the sidecars
Followup to
|
||
|
|
7446dbb3e7 |
refactor(hermes_squid): put sidecars on hermes_proxy instead of admin
Followup to |
||
|
|
b08d25df2f |
fix(hermes_squid): let hermes-gateway reach the new sidecars
The sidecars added in this PR (brencher-state-proxy, docker_socket_proxy)
are reachable on the admin overlay, but hermes-gateway was not.
hermes-gateway was only on the hermes_proxy overlay, so Docker's
embedded DNS could not resolve the new service names from
inside the gateway container. The first
`requests.get("http://brencher-state-proxy:8080/state")` would
have raised `ConnectionError` with "Name or service not known".
NO_PROXY / no_proxy did not list the sidecar hostnames, so even
after fixing overlay membership, any HTTP client inside the
gateway (requests, urllib, etc.) would have routed the call
through squid. Squid would 403 it because the sidecars are not
on the public allowed_domains list, and the overlay DNS names
don't resolve from squid's process.
This commit:
- Adds 'admin' to hermes-gateway's networks list, so the gateway
has L3 reachability to the sidecars on the admin overlay.
- Adds 'brencher-state-proxy' and 'docker_socket_proxy' to both
NO_PROXY and no_proxy so requests skip squid and hit the
sidecars directly.
After this change, code running inside the swarm can do:
import requests
requests.get("http://brencher-state-proxy:8080/state") # public
requests.get("http://brencher-state-proxy:8080/state2") # secondary
os.environ["DOCKER_HOST"] = "tcp://docker_socket_proxy:2375"
and the requests will reach the sidecars without going through squid.
Note: the new overlay membership means hermes-gateway can now L3-reach
ALL services on the admin overlay (registry, gitea, hermes, immich,
gmail_mcp, torrserv_proxy in the brencher stack). That is by design --
the gateway has always been a privileged position in the stack and the
overlay is internal. NO_PROXY is the safety belt that keeps it from
proxying those calls through squid by accident.
Verified:
- YAML parse via PyYAML: ok
- hermes-gateway.networks == ['hermes_proxy', 'admin']
- hermes-gateway.environment.NO_PROXY contains both sidecar names
- hermes-gateway.environment.no_proxy contains both sidecar names
|
||
|
|
0276497a37 |
feat(hermes_squid): drop TLS and secrets from brencher-state-proxy
The previous commit added a sidecar that listened on TLS port 8443
with two docker secrets for the cert and key. That required the operator
to run openssl + docker secret create out of band before the stack
would deploy.
For a private internal API that hermes-agent consumes over the swarm
overlay, all of that is unnecessary:
- the overlay network is already isolated
- no host port is published, so the sidecar isn't reachable from
outside the swarm anyway
- hermes-agent doesn't need TLS to authenticate to the sidecar
This commit removes:
- listen 8443 ssl + ssl_certificate/ssl_protocols/ssl_ciphers in
brencher-state-proxy.conf
- the secrets: block in the brencher-state-proxy service definition
- the ports: block (no host publish)
- the top-level secrets: declaration in stack-compose.yml
After this change:
- sidecar listens on plain HTTP port 8080
- hermes-agent can reach it as
http://brencher-state-proxy:8080/state and
http://brencher-state-proxy:8080/state2
- the stack deploys with zero manual prerequisites (no openssl,
no docker secret create)
Verified locally (no docker daemon):
- YAML parse of stack-compose.yml via PyYAML: ok, no secrets block,
no ports block, extra_hosts and networks still present
- nginx -t: syntax is ok, test is successful
- Live nginx + 9 path tests against two fake upstreams: all 9 pass
(/state -> primary 200, /state2 -> secondary 200, query strings
forwarded to the correct upstream, /state/ /state2/ / /stateX
all 403, /healthz 200)
|
||
|
|
371978aafe |
feat(hermes_squid): add /state2 path for secondary brencher, remove squid whitelist
Two changes: 1) Add /state2 location to brencher-state-proxy.conf that proxies to http://host.docker.internal:5002/state (the secondary brencher instance, reached the same way the brencher container itself reaches it -- via extra_hosts: host.docker.internal:host-gateway). This requires adding the same extra_hosts entry to the brencher-state-proxy service in stack-compose.yml. 2) Remove brencher.rudolf.keenetic.link from squid allowed_domains. The sidecar now handles the only legitimate access path (/state), so squid doesn't need a blanket allow for the host anymore. Any client that wants /state goes through the sidecar; the rest stays blocked at the squid layer (defense in depth). Verified locally (no docker daemon): - nginx -t on brencher-state-proxy.conf: syntax is ok, test successful - Live nginx start against two fake upstreams (primary + secondary): GET /state -> 200 (primary, body contains primary) GET /state2 -> 200 (secondary, body contains secondary) GET /state?x=1 -> 200 (query forwarded to primary, path=/state?x=1) GET /state2?x=1 -> 200 (query forwarded to secondary, path=/state?x=1) GET /state/ -> 403 (trailing slash, exact-match) GET /state2/ -> 403 (trailing slash) GET / -> 403 GET /stateX -> 403 GET /healthz -> 200 - YAML parse of stack-compose.yml via PyYAML: ok, extra_hosts present - All 9 path-locking tests pass |
||
|
|
9b41abbcd3 |
feat(hermes_squid): add brencher-state-proxy sidecar
Adds an nginx sidecar that lets hermes-agent reach https://brencher.rudolf.keenetic.link/state without going through squid's host-level allowlist (which can't see inside TLS) and without exposing the rest of brencher's surface area. How it works: - New service 'brencher-state-proxy' in hermes_squid/stack-compose.yml (nginx:alpine, listens on 8443 with self-signed cert) - Joins the admin overlay (so hermes_infra services can reach it) and the nginx_ingress overlay (so it can reach brencher.rudolf.keenetic.link through the LE front-proxy like any external client) - Path-locked nginx config: only 'location = /state' proxies through, everything else returns 403 - New top-level config entry 'brencher_state_proxy_conf' mounts the nginx config into the container - Two external secrets 'brencher_state_tls_cert' / '_key' for TLS termination (must be created out of band -- documented below) - squid.conf: adds brencher.rudolf.keenetic.link to allowed_domains so the CONNECT itself succeeds for clients that still route via squid Verification done locally (no docker daemon): - YAML parse of stack-compose.yml via PyYAML: ok - nginx -t on brencher-state-proxy.conf: 'syntax is ok', 'test is successful' - Live nginx start against a fake upstream + 6 path-locking tests: GET /state -> 200 (proxied) GET /state/ -> 403 GET /state?foo=bar -> 200 (query string passthrough) GET / -> 403 GET /api/v1/users -> 403 GET /healthz -> 200 - All 6 tests pass Deployment prerequisites (out of scope of this PR, must happen before docker stack deploy succeeds): 1. Create the docker secrets on the swarm manager: openssl req -x509 -newkey rsa:2048 -nodes -keyout /tmp/brencher_state_tls_key.pem -out /tmp/brencher_state_tls_cert.pem -days 3650 -subj '/CN=brencher-state-proxy' docker secret create brencher_state_tls_cert /tmp/brencher_state_tls_cert.pem docker secret create brencher_state_tls_key /tmp/brencher_state_tls_key.pem 2. Ensure the 'nginx_ingress' overlay network exists. It's declared by docker-compose-nginx.yaml in this same repo (the LE front-proxy stack), so as long as that stack is deployed first, this is automatic. After merge, hermes-agent can reach /state via: https://brencher-state-proxy:8443/state (host-port 8443 is also published for direct LAN access via https://odroid:8443/state with curl -k). |
||
|
|
449cbaa316 |
feat(hermes_infra): inline docker_socket_proxy into hermes_squid stack
PR #1 originally shipped docker-socket-proxy as a standalone stack at docker_socket_proxy/stack-compose.yml, exposed via an external overlay network 'socket_proxy' so other stacks could attach to it. Inlining it into hermes_squid/stack-compose.yml as a new service: - same deploy / rollback unit as the rest of hermes_infra - joins the 'admin' overlay network so any hermes_infra service can reach it as 'docker_socket_proxy:2375' - default policy unchanged: read-only PING / EVENTS / VERSION (matches image defaults); every other endpoint explicitly denied - removes the standalone socket_proxy overlay declaration (no longer needed -- consumers attach via 'admin' instead) Brencher config backend/configs/docker_socket_proxy.py is updated in a follow-up commit to point its docker_compose_path at hermes_squid/stack-compose.yml. |
||
|
|
fbed72401a |
feat(docker_socket_proxy): add swarm stack compose for tecnativa/docker-socket-proxy
- HAProxy-based security proxy in front of the host docker socket - binds 2375/tcp on the swarm manager (node.hostname=odroid) - publishes an external 'socket_proxy' overlay network so other stacks (portainer, watchtower, diun) can reach it via DOCKER_HOST=tcp://docker_socket_proxy:2375 - default policy matches upstream: read-only PING/EVENTS/VERSION; everything else denied until explicitly toggled in the environment block |