fix(hermes_infra/gitea): add dedicated act_runner cache-server #34

Merged
admin merged 1 commits from hermes/ansible:fix/gitea-act-runner-cache-server-v2 into master 2026-09-04 08:11:43 +02:00
Contributor

Problem

The built-in act_runner cache server binds to an interface unreachable from per-job container networks (container.network: "" is the default). ACTIONS_CACHE_URL ends up pointing at an unreachable IP, and every CI job burns ~9 minutes on:

::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051
::warning::Failed to save:    reserveCache  failed: connect ETIMEDOUT 192.168.112.21:37051

Measured on admin/brencher#15 (https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16):

Job wall-clock of which dead air on cache
3.12.3 12m 40s ~9m (271s × 2 timeouts)
3.13 11m 13s ~9m
3.14 11m 17s ~9m

So even though python-package.yml enables cache: 'pip', the cache backend is unreachable, jobs hit pip install cold every time, and CI ends up ~4x slower than necessary.

Fix

Run a dedicated act_runner cache-server container on the same overlay network as the runner, and point the runner at it via external_server. Shared secret comes from a Docker secret mounted into both containers and read via ACT_RUNNER_EXTERNAL_SECRET_FILE.

Changes

  • poc/gitea/stack-compose.yml:
    • New cache_server service (command: ["cache-server"]) on the gitea overlay with a cache_data named volume.
    • runner and cache_server both mount the external Docker secret gitea_cache_secret at /run/secrets/gitea_cache_secret and set ACT_RUNNER_EXTERNAL_SECRET_FILE accordingly.
    • Top-level secrets: block declares gitea_cache_secret as external: true, with the create command commented inline next to it.
  • poc/gitea/config.yaml: point cache.{host,port,external_server} at gitea_cache_server:37051; external_secret left empty because the secret is loaded from the file.

Verification

  • Both YAMLs parse (PyYAML).
  • Service name gitea_cache_server resolves on the gitea overlay via Swarm service-name DNS.
  • docker secret inspect gitea_cache_secret works after the one-time create step.

Deploy notes

  1. On the swarm manager, create the secret once:
    echo -n "$(openssl rand -hex 32)" | docker secret create gitea_cache_secret -
    
  2. docker stack deploy -c stack-compose.yml gitea.
  3. Re-run a CI job and confirm cache restore/save log lines no longer ETIMEDOUT.

Related

  • Related: admin/brencher#15 — python-package.yml enables cache: 'pip' and was timing out for the reason this PR fixes.
  • Refs: gitea/actions/runs/16 jobs #248/249/250.

Supersedes

  • Supersedes: admin/ansible#33 — first attempt was based on a stale origin/master and accidentally included hermes_squid/ changes from the actual upstream merge. The current branch is rebased on admin/master (c3b7128) and only touches poc/gitea/.
## Problem The built-in `act_runner` cache server binds to an interface unreachable from per-job container networks (`container.network: ""` is the default). `ACTIONS_CACHE_URL` ends up pointing at an unreachable IP, and every CI job burns ~9 minutes on: ``` ::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051 ::warning::Failed to save: reserveCache failed: connect ETIMEDOUT 192.168.112.21:37051 ``` Measured on `admin/brencher#15` (https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16): | Job | wall-clock | of which dead air on cache | |---------|------------|----------------------------| | 3.12.3 | 12m 40s | ~9m (271s × 2 timeouts) | | 3.13 | 11m 13s | ~9m | | 3.14 | 11m 17s | ~9m | So even though `python-package.yml` enables `cache: 'pip'`, the cache backend is unreachable, jobs hit `pip install` cold every time, and CI ends up ~4x slower than necessary. ## Fix Run a dedicated `act_runner cache-server` container on the same overlay network as the runner, and point the runner at it via `external_server`. Shared secret comes from a Docker secret mounted into both containers and read via `ACT_RUNNER_EXTERNAL_SECRET_FILE`. ## Changes - `poc/gitea/stack-compose.yml`: - New `cache_server` service (`command: ["cache-server"]`) on the `gitea` overlay with a `cache_data` named volume. - `runner` and `cache_server` both mount the external Docker secret `gitea_cache_secret` at `/run/secrets/gitea_cache_secret` and set `ACT_RUNNER_EXTERNAL_SECRET_FILE` accordingly. - Top-level `secrets:` block declares `gitea_cache_secret` as `external: true`, with the create command commented inline next to it. - `poc/gitea/config.yaml`: point `cache.{host,port,external_server}` at `gitea_cache_server:37051`; `external_secret` left empty because the secret is loaded from the file. ## Verification - Both YAMLs parse (PyYAML). - Service name `gitea_cache_server` resolves on the `gitea` overlay via Swarm service-name DNS. - `docker secret inspect gitea_cache_secret` works after the one-time create step. ## Deploy notes 1. On the swarm manager, create the secret once: ``` echo -n "$(openssl rand -hex 32)" | docker secret create gitea_cache_secret - ``` 2. `docker stack deploy -c stack-compose.yml gitea`. 3. Re-run a CI job and confirm cache `restore`/`save` log lines no longer ETIMEDOUT. ## Related - Related: https://gitea.rudolf.keenetic.link/admin/brencher/pulls/15 — `python-package.yml` enables `cache: 'pip'` and was timing out for the reason this PR fixes. - Refs: gitea/actions/runs/16 jobs #248/249/250. ## Supersedes - Supersedes: admin/ansible#33 — first attempt was based on a stale `origin/master` and accidentally included `hermes_squid/` changes from the actual upstream merge. The current branch is rebased on `admin/master` (`c3b7128`) and only touches `poc/gitea/`.
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from b02fc5ecb8 to ee84ae59cd 2026-09-03 21:16:58 +02:00 Compare
admin approved these changes 2026-09-03 21:18:09 +02:00
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from ee84ae59cd to 7b74e58bff 2026-09-03 21:19:13 +02:00 Compare
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from 7b74e58bff to 0d4a162829 2026-09-03 21:24:19 +02:00 Compare
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from 0d4a162829 to 4775ed0cc8 2026-09-03 21:28:32 +02:00 Compare
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from 4775ed0cc8 to 1ec7969009 2026-09-03 21:36:29 +02:00 Compare
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from 1ec7969009 to 2b0ac25a68 2026-09-03 21:46:32 +02:00 Compare
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from 2b0ac25a68 to b6f699617c 2026-09-03 22:45:54 +02:00 Compare
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from b6f699617c to 36de23e858 2026-09-03 23:13:38 +02:00 Compare
hermes added 1 commit 2026-09-03 23:14:03 +02:00
The built-in act_runner cache-server binds to an interface unreachable
from per-job container networks (default container.network: ""), so
ACTIONS_CACHE_URL resolves to 192.168.112.21:37051 and every job burns
~9 minutes on:

    ::warning::Failed to save:   reserveCache failed: connect ETIMEDOUT
    ::warning::Failed to restore: getCacheEntry  failed: connect ETIMEDOUT

Measured on PR #15 in admin/brencher (gitea/actions/runs/16):
each of 3 jobs ~12 min wall-clock, of which 9 min is dead air waiting
for cache timeouts.

Run a dedicated act_runner cache-server on the same overlay network and
point the runner at it via external_server. Shared secret comes from a
Docker secret (gitea_cache_secret) read by both services via
ACT_RUNNER_EXTERNAL_SECRET_FILE.

The image's /usr/local/bin/run.sh entrypoint ignores argv and unconditionally
runs 'act_runner register && act_runner daemon' (the runner path), so
cache_server.service.command must call the act_runner binary directly with
the 'cache-server' subcommand: command: ["act_runner", "cache-server"].

Related: admin/brencher#15
Refs: gitea/actions/runs/16 (jobs #248/249/250)
hermes force-pushed fix/gitea-act-runner-cache-server-v2 from 36de23e858 to 1ba50f8e60 2026-09-03 23:14:03 +02:00 Compare
Author
Contributor

Tracking issue created for this fix and the upstream CI improvement it unblocks:

→ #35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup)

Closes nothing (this PR is what #35 tracks), but linked for cross-reference.

Tracking issue created for this fix and the upstream CI improvement it unblocks: → https://gitea.rudolf.keenetic.link/admin/ansible/issues/35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup) Closes nothing (this PR is what #35 tracks), but linked for cross-reference.
admin merged commit f071c7fb43 into master 2026-09-04 08:11:43 +02:00
admin deleted branch fix/gitea-act-runner-cache-server-v2 2026-09-04 08:11:44 +02:00
Sign in to join this conversation.
No Reviewers
No labels
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/ansible#34