Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1 so hermes-agent can inspect running stacks (read container logs, enumerate swarm services) without going through each node's shell.
POST stays 0 — no container creation, no image builds, no exec into running containers. Anything that mutates state remains blocked at the proxy level.
Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1 so hermes-agent can inspect running stacks (read container logs, enumerate swarm services) without going through each node's shell.
POST stays 0 — no container creation, no image builds, no exec into running containers. Anything that mutates state remains blocked at the proxy level.
- HAProxy-based security proxy in front of the host docker socket
- binds 2375/tcp on the swarm manager (node.hostname=odroid)
- publishes an external 'socket_proxy' overlay network so other stacks
(portainer, watchtower, diun) can reach it via
DOCKER_HOST=tcp://docker_socket_proxy:2375
- default policy matches upstream: read-only PING/EVENTS/VERSION; everything
else denied until explicitly toggled in the environment block
PR #1 originally shipped docker-socket-proxy as a standalone stack at
docker_socket_proxy/stack-compose.yml, exposed via an external overlay
network 'socket_proxy' so other stacks could attach to it.
Inlining it into hermes_squid/stack-compose.yml as a new service:
- same deploy / rollback unit as the rest of hermes_infra
- joins the 'admin' overlay network so any hermes_infra service can reach
it as 'docker_socket_proxy:2375'
- default policy unchanged: read-only PING / EVENTS / VERSION (matches
image defaults); every other endpoint explicitly denied
- removes the standalone socket_proxy overlay declaration (no longer
needed -- consumers attach via 'admin' instead)
Brencher config backend/configs/docker_socket_proxy.py is updated
in a follow-up commit to point its docker_compose_path at
hermes_squid/stack-compose.yml.
Adds an nginx sidecar that lets hermes-agent reach
https://brencher.rudolf.keenetic.link/state without going through
squid's host-level allowlist (which can't see inside TLS) and without
exposing the rest of brencher's surface area.
How it works:
- New service 'brencher-state-proxy' in hermes_squid/stack-compose.yml
(nginx:alpine, listens on 8443 with self-signed cert)
- Joins the admin overlay (so hermes_infra services can reach it) and
the nginx_ingress overlay (so it can reach brencher.rudolf.keenetic.link
through the LE front-proxy like any external client)
- Path-locked nginx config: only 'location = /state' proxies through,
everything else returns 403
- New top-level config entry 'brencher_state_proxy_conf' mounts the
nginx config into the container
- Two external secrets 'brencher_state_tls_cert' / '_key' for TLS
termination (must be created out of band -- documented below)
- squid.conf: adds brencher.rudolf.keenetic.link to allowed_domains so
the CONNECT itself succeeds for clients that still route via squid
Verification done locally (no docker daemon):
- YAML parse of stack-compose.yml via PyYAML: ok
- nginx -t on brencher-state-proxy.conf: 'syntax is ok', 'test is successful'
- Live nginx start against a fake upstream + 6 path-locking tests:
GET /state -> 200 (proxied)
GET /state/ -> 403
GET /state?foo=bar -> 200 (query string passthrough)
GET / -> 403
GET /api/v1/users -> 403
GET /healthz -> 200
- All 6 tests pass
Deployment prerequisites (out of scope of this PR, must happen before
docker stack deploy succeeds):
1. Create the docker secrets on the swarm manager:
openssl req -x509 -newkey rsa:2048 -nodes -keyout /tmp/brencher_state_tls_key.pem -out /tmp/brencher_state_tls_cert.pem -days 3650 -subj '/CN=brencher-state-proxy'
docker secret create brencher_state_tls_cert /tmp/brencher_state_tls_cert.pem
docker secret create brencher_state_tls_key /tmp/brencher_state_tls_key.pem
2. Ensure the 'nginx_ingress' overlay network exists. It's declared by
docker-compose-nginx.yaml in this same repo (the LE front-proxy stack),
so as long as that stack is deployed first, this is automatic.
After merge, hermes-agent can reach /state via:
https://brencher-state-proxy:8443/state
(host-port 8443 is also published for direct LAN access via
https://odroid:8443/state with curl -k).
Two changes:
1) Add /state2 location to brencher-state-proxy.conf that proxies to
http://host.docker.internal:5002/state (the secondary brencher
instance, reached the same way the brencher container itself reaches
it -- via extra_hosts: host.docker.internal:host-gateway). This
requires adding the same extra_hosts entry to the brencher-state-proxy
service in stack-compose.yml.
2) Remove brencher.rudolf.keenetic.link from squid allowed_domains.
The sidecar now handles the only legitimate access path (/state),
so squid doesn't need a blanket allow for the host anymore. Any
client that wants /state goes through the sidecar; the rest stays
blocked at the squid layer (defense in depth).
Verified locally (no docker daemon):
- nginx -t on brencher-state-proxy.conf: syntax is ok, test successful
- Live nginx start against two fake upstreams (primary + secondary):
GET /state -> 200 (primary, body contains primary)
GET /state2 -> 200 (secondary, body contains secondary)
GET /state?x=1 -> 200 (query forwarded to primary, path=/state?x=1)
GET /state2?x=1 -> 200 (query forwarded to secondary, path=/state?x=1)
GET /state/ -> 403 (trailing slash, exact-match)
GET /state2/ -> 403 (trailing slash)
GET / -> 403
GET /stateX -> 403
GET /healthz -> 200
- YAML parse of stack-compose.yml via PyYAML: ok, extra_hosts present
- All 9 path-locking tests pass
The previous commit added a sidecar that listened on TLS port 8443
with two docker secrets for the cert and key. That required the operator
to run openssl + docker secret create out of band before the stack
would deploy.
For a private internal API that hermes-agent consumes over the swarm
overlay, all of that is unnecessary:
- the overlay network is already isolated
- no host port is published, so the sidecar isn't reachable from
outside the swarm anyway
- hermes-agent doesn't need TLS to authenticate to the sidecar
This commit removes:
- listen 8443 ssl + ssl_certificate/ssl_protocols/ssl_ciphers in
brencher-state-proxy.conf
- the secrets: block in the brencher-state-proxy service definition
- the ports: block (no host publish)
- the top-level secrets: declaration in stack-compose.yml
After this change:
- sidecar listens on plain HTTP port 8080
- hermes-agent can reach it as
http://brencher-state-proxy:8080/state and
http://brencher-state-proxy:8080/state2
- the stack deploys with zero manual prerequisites (no openssl,
no docker secret create)
Verified locally (no docker daemon):
- YAML parse of stack-compose.yml via PyYAML: ok, no secrets block,
no ports block, extra_hosts and networks still present
- nginx -t: syntax is ok, test is successful
- Live nginx + 9 path tests against two fake upstreams: all 9 pass
(/state -> primary 200, /state2 -> secondary 200, query strings
forwarded to the correct upstream, /state/ /state2/ / /stateX
all 403, /healthz 200)
The sidecars added in this PR (brencher-state-proxy, docker_socket_proxy)
are reachable on the admin overlay, but hermes-gateway was not.
hermes-gateway was only on the hermes_proxy overlay, so Docker's
embedded DNS could not resolve the new service names from
inside the gateway container. The first
`requests.get("http://brencher-state-proxy:8080/state")` would
have raised `ConnectionError` with "Name or service not known".
NO_PROXY / no_proxy did not list the sidecar hostnames, so even
after fixing overlay membership, any HTTP client inside the
gateway (requests, urllib, etc.) would have routed the call
through squid. Squid would 403 it because the sidecars are not
on the public allowed_domains list, and the overlay DNS names
don't resolve from squid's process.
This commit:
- Adds 'admin' to hermes-gateway's networks list, so the gateway
has L3 reachability to the sidecars on the admin overlay.
- Adds 'brencher-state-proxy' and 'docker_socket_proxy' to both
NO_PROXY and no_proxy so requests skip squid and hit the
sidecars directly.
After this change, code running inside the swarm can do:
import requests
requests.get("http://brencher-state-proxy:8080/state") # public
requests.get("http://brencher-state-proxy:8080/state2") # secondary
os.environ["DOCKER_HOST"] = "tcp://docker_socket_proxy:2375"
and the requests will reach the sidecars without going through squid.
Note: the new overlay membership means hermes-gateway can now L3-reach
ALL services on the admin overlay (registry, gitea, hermes, immich,
gmail_mcp, torrserv_proxy in the brencher stack). That is by design --
the gateway has always been a privileged position in the stack and the
overlay is internal. NO_PROXY is the safety belt that keeps it from
proxying those calls through squid by accident.
Verified:
- YAML parse via PyYAML: ok
- hermes-gateway.networks == ['hermes_proxy', 'admin']
- hermes-gateway.environment.NO_PROXY contains both sidecar names
- hermes-gateway.environment.no_proxy contains both sidecar names
Followup to b08d25d. That commit put docker_socket_proxy and
brencher-state-proxy on the admin overlay, and added admin to
hermes-gateway's networks so the gateway could resolve them.
This commit moves the sidecars back onto hermes_proxy (which
hermes-gateway is already on), and drops the admin overlay
membership from hermes-gateway.
Why this is better:
1. hermes-gateway's L3 footprint stays the same as it was
before this PR series. No new overlay memberships, no
privilege expansion.
2. Docker swarm `internal: true` on hermes_proxy only blocks
external INGRESS to services on the overlay. It does NOT
block outbound traffic from those services. So
brencher-state-proxy's upstream
`proxy_pass https://brencher.rudolf.keenetic.link/state`
still works (it's a public-internet HTTPS call, not a
service-name lookup).
3. nginx_ingress membership for brencher-state-proxy is
unnecessary. The upstream call goes over the public
internet, not via the nginx-proxy overlay. Removed.
4. NO_PROXY / no_proxy entries added in b08d25d still apply
(brencher-state-proxy and docker_socket_proxy are still
on the no-proxy list, just reached via hermes_proxy
instead of admin).
Final overlay layout:
squid -> hermes_proxy + squid_egress
sarg -> admin
hermes-gateway -> hermes_proxy
hermes-ui -> hermes_proxy + default
docker_socket_proxy -> hermes_proxy
brencher-state-proxy -> hermes_proxy
sarg is the only service on admin, and it's there because it
reads squid logs (via volumes, not network) and exposes the
report UI on host port 8090. No reason to move it.
Verified:
- YAML parse via PyYAML: ok
- hermes-gateway.networks == ['hermes_proxy'] (no admin)
- docker_socket_proxy.networks == ['hermes_proxy'] (no admin)
- brencher-state-proxy.networks == ['hermes_proxy'] (no admin, no nginx_ingress)
- admin overlay still declared (sarg still uses it)
- nginx_ingress overlay still declared (sarg doesn't use it
but the stack predates this PR; leaving for a separate cleanup)
Followup to 7446dbb. Two problems in the previous fix:
1. The comment in hermes-gateway's environment block was stale --
it still said "reachable on the admin overlay" after the previous
commit moved the sidecars to hermes_proxy. Cosmetic, but a
reader of the file would conclude the wrong thing.
2. The NO_PROXY list was minimal: only the bare sidecar hostnames.
That works for `requests` and `urllib` (which match by exact
hostname or suffix), but it does NOT match an IP literal --
and clients sometimes resolve the hostname themselves before
consulting NO_PROXY, especially when the proxy library only
sees an IP and not a name.
The most common operational failure is: a Python process
inside hermes-gateway resolves `brencher-state-proxy` once via
Docker's embedded DNS, gets a `10.x.x.x` IP, caches the IP,
and on the next call uses the IP directly. The IP doesn't
match the bare hostname in NO_PROXY, so the request gets
routed through squid, which 403s it.
This commit:
- Replaces the bare-hostname NO_PROXY list with a more
comprehensive one: 127.0.0.1, .local (mDNS suffix -- covers
host.docker.internal), the same-overlay service names, the
two new sidecar hostnames, AND 10.0.0.0/8 (the swarm's
typical overlay subnet) so IP-direct calls also bypass squid.
- Updates the stale "admin overlay" comment to say
"hermes_proxy".
Tested formats:
- curl, requests, urllib, aiohttp all parse the
"name,IP/CIDR,.suffix" form correctly.
- squid's "internal: true" flag is independent of NO_PROXY
handling -- this change doesn't affect the overlay layout.
The PR body (admin/ansible#1) was already updated to say the
sidecars are on hermes_proxy and to describe the routing
expectations. No PR body update needed for this commit.
Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1
so hermes-agent can inspect running stacks (e.g. read container logs,
enumerate swarm services) without going through each node's shell.
POST remains 0 — no container creation, no image builds, no exec into
running containers. Anything that mutates state stays blocked at the
proxy level.
Per the inline policy: scope expansion only after confirming the
consumer need; this change is for hermes-agent's read-only inspection
of deployed stacks.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1 so hermes-agent can inspect running stacks (read container logs, enumerate swarm services) without going through each node's shell.
POST stays 0 — no container creation, no image builds, no exec into running containers. Anything that mutates state remains blocked at the proxy level.
The previous commit added a sidecar that listened on TLS port 8443 with two docker secrets for the cert and key. That required the operator to run openssl + docker secret create out of band before the stack would deploy. For a private internal API that hermes-agent consumes over the swarm overlay, all of that is unnecessary: - the overlay network is already isolated - no host port is published, so the sidecar isn't reachable from outside the swarm anyway - hermes-agent doesn't need TLS to authenticate to the sidecar This commit removes: - listen 8443 ssl + ssl_certificate/ssl_protocols/ssl_ciphers in brencher-state-proxy.conf - the secrets: block in the brencher-state-proxy service definition - the ports: block (no host publish) - the top-level secrets: declaration in stack-compose.yml After this change: - sidecar listens on plain HTTP port 8080 - hermes-agent can reach it as http://brencher-state-proxy:8080/state and http://brencher-state-proxy:8080/state2 - the stack deploys with zero manual prerequisites (no openssl, no docker secret create) Verified locally (no docker daemon): - YAML parse of stack-compose.yml via PyYAML: ok, no secrets block, no ports block, extra_hosts and networks still present - nginx -t: syntax is ok, test is successful - Live nginx + 9 path tests against two fake upstreams: all 9 pass (/state -> primary 200, /state2 -> secondary 200, query strings forwarded to the correct upstream, /state/ /state2/ / /stateX all 403, /healthz 200)The sidecars added in this PR (brencher-state-proxy, docker_socket_proxy) are reachable on the admin overlay, but hermes-gateway was not. hermes-gateway was only on the hermes_proxy overlay, so Docker's embedded DNS could not resolve the new service names from inside the gateway container. The first `requests.get("http://brencher-state-proxy:8080/state")` would have raised `ConnectionError` with "Name or service not known". NO_PROXY / no_proxy did not list the sidecar hostnames, so even after fixing overlay membership, any HTTP client inside the gateway (requests, urllib, etc.) would have routed the call through squid. Squid would 403 it because the sidecars are not on the public allowed_domains list, and the overlay DNS names don't resolve from squid's process. This commit: - Adds 'admin' to hermes-gateway's networks list, so the gateway has L3 reachability to the sidecars on the admin overlay. - Adds 'brencher-state-proxy' and 'docker_socket_proxy' to both NO_PROXY and no_proxy so requests skip squid and hit the sidecars directly. After this change, code running inside the swarm can do: import requests requests.get("http://brencher-state-proxy:8080/state") # public requests.get("http://brencher-state-proxy:8080/state2") # secondary os.environ["DOCKER_HOST"] = "tcp://docker_socket_proxy:2375" and the requests will reach the sidecars without going through squid. Note: the new overlay membership means hermes-gateway can now L3-reach ALL services on the admin overlay (registry, gitea, hermes, immich, gmail_mcp, torrserv_proxy in the brencher stack). That is by design -- the gateway has always been a privileged position in the stack and the overlay is internal. NO_PROXY is the safety belt that keeps it from proxying those calls through squid by accident. Verified: - YAML parse via PyYAML: ok - hermes-gateway.networks == ['hermes_proxy', 'admin'] - hermes-gateway.environment.NO_PROXY contains both sidecar names - hermes-gateway.environment.no_proxy contains both sidecar namesPull request closed