fix(hermes_infra/gitea): add dedicated act_runner cache-server #33

Closed
hermes wants to merge 2 commits from hermes/ansible:fix/gitea-act-runner-cache-server into master
Contributor

Problem

The built-in act_runner cache server binds to an interface unreachable from per-job container networks (container.network: "" is the default). ACTIONS_CACHE_URL ends up pointing at an unreachable IP, and every CI job burns ~9 minutes on:

::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051
::warning::Failed to save:    reserveCache  failed: connect ETIMEDOUT 192.168.112.21:37051

Measured on admin/brencher#15 (https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16):

Job wall-clock of which dead air on cache
3.12.3 12m 40s ~9m (271s × 2 timeouts)
3.13 11m 13s ~9m
3.14 11m 17s ~9m

So even though python-package.yml enables cache: 'pip', the cache backend is unreachable, jobs hit pip install cold every time, and CI ends up ~4x slower than necessary.

Fix

Run a dedicated act_runner cache-server container on the same overlay network as the runner, and point the runner at it via external_server. Secret shared via ACT_RUNNER_EXTERNAL_SECRET env on both services (placeholder CHANGE_ME_SHARED_SECRET — replace with a real random value before deploy via Docker secret or env file).

Changes

  • poc/gitea/stack-compose.yml: add cache_server service (command: ["cache-server"]) on the gitea overlay with cache_data named volume; add ACT_RUNNER_EXTERNAL_SECRET env to runner.
  • poc/gitea/config.yaml: point cache.{host,port,external_server} at the new service, set external_secret.

Verification

  • Both YAMLs parse (PyYAML).
  • Service name gitea_cache_server resolves on the gitea overlay via Swarm service-name DNS.
  • The runner container reads the same external_secret from env as the cache-server reads from its env (deploy must keep these in sync).

Deploy notes

  1. Generate a real shared secret: openssl rand -hex 32.
  2. Replace CHANGE_ME_SHARED_SECRET in both places. The config is bound as a Swarm config; the cache-server env should be a Docker secret or host env, not the config.
  3. docker stack deploy -c stack-compose.yml gitea.
  4. Re-run a CI job and confirm cache restore/save log lines no longer ETIMEDOUT.

Related

  • Related: admin/brencher#15 — python-package.yml enables cache: 'pip' and was timing out for the reason this PR fixes.
  • Refs: gitea/actions/runs/16 jobs #248/249/250.
## Problem The built-in `act_runner` cache server binds to an interface unreachable from per-job container networks (`container.network: ""` is the default). `ACTIONS_CACHE_URL` ends up pointing at an unreachable IP, and every CI job burns ~9 minutes on: ``` ::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051 ::warning::Failed to save: reserveCache failed: connect ETIMEDOUT 192.168.112.21:37051 ``` Measured on `admin/brencher#15` (https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16): | Job | wall-clock | of which dead air on cache | |---------|------------|----------------------------| | 3.12.3 | 12m 40s | ~9m (271s × 2 timeouts) | | 3.13 | 11m 13s | ~9m | | 3.14 | 11m 17s | ~9m | So even though `python-package.yml` enables `cache: 'pip'`, the cache backend is unreachable, jobs hit `pip install` cold every time, and CI ends up ~4x slower than necessary. ## Fix Run a dedicated `act_runner cache-server` container on the same overlay network as the runner, and point the runner at it via `external_server`. Secret shared via `ACT_RUNNER_EXTERNAL_SECRET` env on both services (placeholder `CHANGE_ME_SHARED_SECRET` — replace with a real random value before deploy via Docker secret or env file). ## Changes - `poc/gitea/stack-compose.yml`: add `cache_server` service (`command: ["cache-server"]`) on the `gitea` overlay with `cache_data` named volume; add `ACT_RUNNER_EXTERNAL_SECRET` env to `runner`. - `poc/gitea/config.yaml`: point `cache.{host,port,external_server}` at the new service, set `external_secret`. ## Verification - Both YAMLs parse (PyYAML). - Service name `gitea_cache_server` resolves on the `gitea` overlay via Swarm service-name DNS. - The runner container reads the same `external_secret` from env as the cache-server reads from its env (deploy must keep these in sync). ## Deploy notes 1. Generate a real shared secret: `openssl rand -hex 32`. 2. Replace `CHANGE_ME_SHARED_SECRET` in both places. The config is bound as a Swarm config; the cache-server env should be a Docker secret or host env, not the config. 3. `docker stack deploy -c stack-compose.yml gitea`. 4. Re-run a CI job and confirm cache `restore`/`save` log lines no longer ETIMEDOUT. ## Related - Related: https://gitea.rudolf.keenetic.link/admin/brencher/pulls/15 — `python-package.yml` enables `cache: 'pip'` and was timing out for the reason this PR fixes. - Refs: gitea/actions/runs/16 jobs #248/249/250.
hermes added 2 commits 2026-09-03 20:42:07 +02:00
Drop the brencher-state-proxy sidecar from hermes_infra -- it is
no longer needed. The sidecar existed to relay /state from
brencher.rudolf.keenetic.link into the admin overlay; that relay is
unnecessary since brencher ingress handles the path itself.

Changes vs the brencher-state-proxy commit that was merged in #4:
- delete hermes_squid/brencher-state-proxy.conf
- remove the brencher-state-proxy: service block from
  hermes_squid/stack-compose.yml
- remove the brencher_state_proxy_conf: top-level config object
- drop brencher-state-proxy from the NO_PROXY / no_proxy lists
- update the surrounding comments
The built-in act_runner cache-server binds to an interface unreachable
from per-job container networks (default container.network: ""), so
ACTIONS_CACHE_URL resolves to 192.168.112.21:37051 and every job burns
~9 minutes on:

    ::warning::Failed to save:   reserveCache failed: connect ETIMEDOUT
    ::warning::Failed to restore: getCacheEntry  failed: connect ETIMEDOUT

Measured on PR #15 in admin/brencher (gitea/actions/runs/16):
each of 3 jobs ≈12 min wall-clock, of which 9 min is dead air waiting
for cache timeouts.

Run a dedicated act_runner cache-server on the same overlay network and
point the runner at it via external_server. secret shared via
ACT_RUNNER_EXTERNAL_SECRET env on both services.

Related: admin/brencher#15 (whose python-package.yml enables cache:'pip'
but currently gets no benefit because the cache backend is unreachable).

Refs: gitea/actions/runs/16 (job #248 logs)
hermes closed this pull request 2026-09-03 20:51:43 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/ansible#33