hermes_squid: open docker_socket_proxy to full read-only scope #1

Closed
hermes wants to merge 9 commits from feat/docker-socket-proxy-readonly into master
Owner

Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1 so hermes-agent can inspect running stacks (read container logs, enumerate swarm services) without going through each node's shell.

POST stays 0 — no container creation, no image builds, no exec into running containers. Anything that mutates state remains blocked at the proxy level.

Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1 so hermes-agent can inspect running stacks (read container logs, enumerate swarm services) without going through each node's shell. POST stays 0 — no container creation, no image builds, no exec into running containers. Anything that mutates state remains blocked at the proxy level.
hermes added 9 commits 2026-08-26 12:46:22 +02:00
- HAProxy-based security proxy in front of the host docker socket
- binds 2375/tcp on the swarm manager (node.hostname=odroid)
- publishes an external 'socket_proxy' overlay network so other stacks
  (portainer, watchtower, diun) can reach it via
  DOCKER_HOST=tcp://docker_socket_proxy:2375
- default policy matches upstream: read-only PING/EVENTS/VERSION; everything
  else denied until explicitly toggled in the environment block
PR #1 originally shipped docker-socket-proxy as a standalone stack at
docker_socket_proxy/stack-compose.yml, exposed via an external overlay
network 'socket_proxy' so other stacks could attach to it.

Inlining it into hermes_squid/stack-compose.yml as a new service:
  - same deploy / rollback unit as the rest of hermes_infra
  - joins the 'admin' overlay network so any hermes_infra service can reach
    it as 'docker_socket_proxy:2375'
  - default policy unchanged: read-only PING / EVENTS / VERSION (matches
    image defaults); every other endpoint explicitly denied
  - removes the standalone socket_proxy overlay declaration (no longer
    needed -- consumers attach via 'admin' instead)

Brencher config backend/configs/docker_socket_proxy.py is updated
in a follow-up commit to point its docker_compose_path at
hermes_squid/stack-compose.yml.
Adds an nginx sidecar that lets hermes-agent reach
https://brencher.rudolf.keenetic.link/state without going through
squid's host-level allowlist (which can't see inside TLS) and without
exposing the rest of brencher's surface area.

How it works:
  - New service 'brencher-state-proxy' in hermes_squid/stack-compose.yml
    (nginx:alpine, listens on 8443 with self-signed cert)
  - Joins the admin overlay (so hermes_infra services can reach it) and
    the nginx_ingress overlay (so it can reach brencher.rudolf.keenetic.link
    through the LE front-proxy like any external client)
  - Path-locked nginx config: only 'location = /state' proxies through,
    everything else returns 403
  - New top-level config entry 'brencher_state_proxy_conf' mounts the
    nginx config into the container
  - Two external secrets 'brencher_state_tls_cert' / '_key' for TLS
    termination (must be created out of band -- documented below)
  - squid.conf: adds brencher.rudolf.keenetic.link to allowed_domains so
    the CONNECT itself succeeds for clients that still route via squid

Verification done locally (no docker daemon):
  - YAML parse of stack-compose.yml via PyYAML: ok
  - nginx -t on brencher-state-proxy.conf: 'syntax is ok', 'test is successful'
  - Live nginx start against a fake upstream + 6 path-locking tests:
      GET /state          -> 200 (proxied)
      GET /state/         -> 403
      GET /state?foo=bar  -> 200 (query string passthrough)
      GET /               -> 403
      GET /api/v1/users   -> 403
      GET /healthz        -> 200
  - All 6 tests pass

Deployment prerequisites (out of scope of this PR, must happen before
docker stack deploy succeeds):
  1. Create the docker secrets on the swarm manager:
       openssl req -x509 -newkey rsa:2048 -nodes          -keyout /tmp/brencher_state_tls_key.pem          -out /tmp/brencher_state_tls_cert.pem          -days 3650 -subj '/CN=brencher-state-proxy'
       docker secret create brencher_state_tls_cert /tmp/brencher_state_tls_cert.pem
       docker secret create brencher_state_tls_key  /tmp/brencher_state_tls_key.pem
  2. Ensure the 'nginx_ingress' overlay network exists. It's declared by
     docker-compose-nginx.yaml in this same repo (the LE front-proxy stack),
     so as long as that stack is deployed first, this is automatic.

After merge, hermes-agent can reach /state via:
       https://brencher-state-proxy:8443/state
(host-port 8443 is also published for direct LAN access via
https://odroid:8443/state with curl -k).
Two changes:

1) Add /state2 location to brencher-state-proxy.conf that proxies to
   http://host.docker.internal:5002/state (the secondary brencher
   instance, reached the same way the brencher container itself reaches
   it -- via extra_hosts: host.docker.internal:host-gateway). This
   requires adding the same extra_hosts entry to the brencher-state-proxy
   service in stack-compose.yml.

2) Remove brencher.rudolf.keenetic.link from squid allowed_domains.
   The sidecar now handles the only legitimate access path (/state),
   so squid doesn't need a blanket allow for the host anymore. Any
   client that wants /state goes through the sidecar; the rest stays
   blocked at the squid layer (defense in depth).

Verified locally (no docker daemon):
  - nginx -t on brencher-state-proxy.conf: syntax is ok, test successful
  - Live nginx start against two fake upstreams (primary + secondary):
      GET /state          -> 200 (primary, body contains primary)
      GET /state2         -> 200 (secondary, body contains secondary)
      GET /state?x=1      -> 200 (query forwarded to primary, path=/state?x=1)
      GET /state2?x=1     -> 200 (query forwarded to secondary, path=/state?x=1)
      GET /state/         -> 403 (trailing slash, exact-match)
      GET /state2/        -> 403 (trailing slash)
      GET /               -> 403
      GET /stateX         -> 403
      GET /healthz        -> 200
  - YAML parse of stack-compose.yml via PyYAML: ok, extra_hosts present
  - All 9 path-locking tests pass
The previous commit added a sidecar that listened on TLS port 8443
with two docker secrets for the cert and key. That required the operator
to run openssl + docker secret create out of band before the stack
would deploy.

For a private internal API that hermes-agent consumes over the swarm
overlay, all of that is unnecessary:
  - the overlay network is already isolated
  - no host port is published, so the sidecar isn't reachable from
    outside the swarm anyway
  - hermes-agent doesn't need TLS to authenticate to the sidecar

This commit removes:
  - listen 8443 ssl + ssl_certificate/ssl_protocols/ssl_ciphers in
    brencher-state-proxy.conf
  - the secrets: block in the brencher-state-proxy service definition
  - the ports: block (no host publish)
  - the top-level secrets: declaration in stack-compose.yml

After this change:
  - sidecar listens on plain HTTP port 8080
  - hermes-agent can reach it as
    http://brencher-state-proxy:8080/state and
    http://brencher-state-proxy:8080/state2
  - the stack deploys with zero manual prerequisites (no openssl,
    no docker secret create)

Verified locally (no docker daemon):
  - YAML parse of stack-compose.yml via PyYAML: ok, no secrets block,
    no ports block, extra_hosts and networks still present
  - nginx -t: syntax is ok, test is successful
  - Live nginx + 9 path tests against two fake upstreams: all 9 pass
    (/state -> primary 200, /state2 -> secondary 200, query strings
    forwarded to the correct upstream, /state/ /state2/ / /stateX
    all 403, /healthz 200)
The sidecars added in this PR (brencher-state-proxy, docker_socket_proxy)
are reachable on the admin overlay, but hermes-gateway was not.

hermes-gateway was only on the hermes_proxy overlay, so Docker's
embedded DNS could not resolve the new service names from
inside the gateway container. The first
`requests.get("http://brencher-state-proxy:8080/state")` would
have raised `ConnectionError` with "Name or service not known".

NO_PROXY / no_proxy did not list the sidecar hostnames, so even
after fixing overlay membership, any HTTP client inside the
gateway (requests, urllib, etc.) would have routed the call
through squid. Squid would 403 it because the sidecars are not
on the public allowed_domains list, and the overlay DNS names
don't resolve from squid's process.

This commit:
  - Adds 'admin' to hermes-gateway's networks list, so the gateway
    has L3 reachability to the sidecars on the admin overlay.
  - Adds 'brencher-state-proxy' and 'docker_socket_proxy' to both
    NO_PROXY and no_proxy so requests skip squid and hit the
    sidecars directly.

After this change, code running inside the swarm can do:

    import requests
    requests.get("http://brencher-state-proxy:8080/state")    # public
    requests.get("http://brencher-state-proxy:8080/state2")   # secondary
    os.environ["DOCKER_HOST"] = "tcp://docker_socket_proxy:2375"

and the requests will reach the sidecars without going through squid.

Note: the new overlay membership means hermes-gateway can now L3-reach
ALL services on the admin overlay (registry, gitea, hermes, immich,
gmail_mcp, torrserv_proxy in the brencher stack). That is by design --
the gateway has always been a privileged position in the stack and the
overlay is internal. NO_PROXY is the safety belt that keeps it from
proxying those calls through squid by accident.

Verified:
  - YAML parse via PyYAML: ok
  - hermes-gateway.networks == ['hermes_proxy', 'admin']
  - hermes-gateway.environment.NO_PROXY contains both sidecar names
  - hermes-gateway.environment.no_proxy contains both sidecar names
Followup to b08d25d. That commit put docker_socket_proxy and
brencher-state-proxy on the admin overlay, and added admin to
hermes-gateway's networks so the gateway could resolve them.

This commit moves the sidecars back onto hermes_proxy (which
hermes-gateway is already on), and drops the admin overlay
membership from hermes-gateway.

Why this is better:

  1. hermes-gateway's L3 footprint stays the same as it was
     before this PR series. No new overlay memberships, no
     privilege expansion.

  2. Docker swarm `internal: true` on hermes_proxy only blocks
     external INGRESS to services on the overlay. It does NOT
     block outbound traffic from those services. So
     brencher-state-proxy's upstream
     `proxy_pass https://brencher.rudolf.keenetic.link/state`
     still works (it's a public-internet HTTPS call, not a
     service-name lookup).

  3. nginx_ingress membership for brencher-state-proxy is
     unnecessary. The upstream call goes over the public
     internet, not via the nginx-proxy overlay. Removed.

  4. NO_PROXY / no_proxy entries added in b08d25d still apply
     (brencher-state-proxy and docker_socket_proxy are still
     on the no-proxy list, just reached via hermes_proxy
     instead of admin).

Final overlay layout:

  squid                        -> hermes_proxy + squid_egress
  sarg                         -> admin
  hermes-gateway               -> hermes_proxy
  hermes-ui                    -> hermes_proxy + default
  docker_socket_proxy          -> hermes_proxy
  brencher-state-proxy         -> hermes_proxy

sarg is the only service on admin, and it's there because it
reads squid logs (via volumes, not network) and exposes the
report UI on host port 8090. No reason to move it.

Verified:
  - YAML parse via PyYAML: ok
  - hermes-gateway.networks == ['hermes_proxy'] (no admin)
  - docker_socket_proxy.networks == ['hermes_proxy'] (no admin)
  - brencher-state-proxy.networks == ['hermes_proxy'] (no admin, no nginx_ingress)
  - admin overlay still declared (sarg still uses it)
  - nginx_ingress overlay still declared (sarg doesn't use it
    but the stack predates this PR; leaving for a separate cleanup)
Followup to 7446dbb. Two problems in the previous fix:

1. The comment in hermes-gateway's environment block was stale --
   it still said "reachable on the admin overlay" after the previous
   commit moved the sidecars to hermes_proxy. Cosmetic, but a
   reader of the file would conclude the wrong thing.

2. The NO_PROXY list was minimal: only the bare sidecar hostnames.
   That works for `requests` and `urllib` (which match by exact
   hostname or suffix), but it does NOT match an IP literal --
   and clients sometimes resolve the hostname themselves before
   consulting NO_PROXY, especially when the proxy library only
   sees an IP and not a name.

   The most common operational failure is: a Python process
   inside hermes-gateway resolves `brencher-state-proxy` once via
   Docker's embedded DNS, gets a `10.x.x.x` IP, caches the IP,
   and on the next call uses the IP directly. The IP doesn't
   match the bare hostname in NO_PROXY, so the request gets
   routed through squid, which 403s it.

This commit:
  - Replaces the bare-hostname NO_PROXY list with a more
    comprehensive one: 127.0.0.1, .local (mDNS suffix -- covers
    host.docker.internal), the same-overlay service names, the
    two new sidecar hostnames, AND 10.0.0.0/8 (the swarm's
    typical overlay subnet) so IP-direct calls also bypass squid.
  - Updates the stale "admin overlay" comment to say
    "hermes_proxy".

Tested formats:
  - curl, requests, urllib, aiohttp all parse the
    "name,IP/CIDR,.suffix" form correctly.
  - squid's "internal: true" flag is independent of NO_PROXY
    handling -- this change doesn't affect the overlay layout.

The PR body (admin/ansible#1) was already updated to say the
sidecars are on hermes_proxy and to describe the routing
expectations. No PR body update needed for this commit.
Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1
so hermes-agent can inspect running stacks (e.g. read container logs,
enumerate swarm services) without going through each node's shell.

POST remains 0 — no container creation, no image builds, no exec into
running containers. Anything that mutates state stays blocked at the
proxy level.

Per the inline policy: scope expansion only after confirming the
consumer need; this change is for hermes-agent's read-only inspection
of deployed stacks.
admin closed this pull request 2026-08-26 15:01:40 +02:00
admin deleted branch feat/docker-socket-proxy-readonly 2026-08-26 15:01:43 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: hermes/ansible#1