The built-in act_runner cache server binds to an interface unreachable from per-job container networks (container.network: "" is the default). ACTIONS_CACHE_URL ends up pointing at an unreachable IP, and every CI job burns ~9 minutes on:
::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051
::warning::Failed to save: reserveCache failed: connect ETIMEDOUT 192.168.112.21:37051
So even though python-package.yml enables cache: 'pip', the cache backend is unreachable, jobs hit pip install cold every time, and CI ends up ~4x slower than necessary.
Fix
Run a dedicated act_runner cache-server container on the same overlay network as the runner, and point the runner at it via external_server. Shared secret comes from a Docker secret mounted into both containers and read via ACT_RUNNER_EXTERNAL_SECRET_FILE.
Changes
poc/gitea/stack-compose.yml:
New cache_server service (command: ["cache-server"]) on the gitea overlay with a cache_data named volume.
runner and cache_server both mount the external Docker secret gitea_cache_secret at /run/secrets/gitea_cache_secret and set ACT_RUNNER_EXTERNAL_SECRET_FILE accordingly.
Top-level secrets: block declares gitea_cache_secret as external: true, with the create command commented inline next to it.
poc/gitea/config.yaml: point cache.{host,port,external_server} at gitea_cache_server:37051; external_secret left empty because the secret is loaded from the file.
Verification
Both YAMLs parse (PyYAML).
Service name gitea_cache_server resolves on the gitea overlay via Swarm service-name DNS.
docker secret inspect gitea_cache_secret works after the one-time create step.
Re-run a CI job and confirm cache restore/save log lines no longer ETIMEDOUT.
Related
Related: admin/brencher#15 — python-package.yml enables cache: 'pip' and was timing out for the reason this PR fixes.
Refs: gitea/actions/runs/16 jobs #248/249/250.
Supersedes
Supersedes: admin/ansible#33 — first attempt was based on a stale origin/master and accidentally included hermes_squid/ changes from the actual upstream merge. The current branch is rebased on admin/master (c3b7128) and only touches poc/gitea/.
## Problem
The built-in `act_runner` cache server binds to an interface unreachable from per-job container networks (`container.network: ""` is the default). `ACTIONS_CACHE_URL` ends up pointing at an unreachable IP, and every CI job burns ~9 minutes on:
```
::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051
::warning::Failed to save: reserveCache failed: connect ETIMEDOUT 192.168.112.21:37051
```
Measured on `admin/brencher#15` (https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16):
| Job | wall-clock | of which dead air on cache |
|---------|------------|----------------------------|
| 3.12.3 | 12m 40s | ~9m (271s × 2 timeouts) |
| 3.13 | 11m 13s | ~9m |
| 3.14 | 11m 17s | ~9m |
So even though `python-package.yml` enables `cache: 'pip'`, the cache backend is unreachable, jobs hit `pip install` cold every time, and CI ends up ~4x slower than necessary.
## Fix
Run a dedicated `act_runner cache-server` container on the same overlay network as the runner, and point the runner at it via `external_server`. Shared secret comes from a Docker secret mounted into both containers and read via `ACT_RUNNER_EXTERNAL_SECRET_FILE`.
## Changes
- `poc/gitea/stack-compose.yml`:
- New `cache_server` service (`command: ["cache-server"]`) on the `gitea` overlay with a `cache_data` named volume.
- `runner` and `cache_server` both mount the external Docker secret `gitea_cache_secret` at `/run/secrets/gitea_cache_secret` and set `ACT_RUNNER_EXTERNAL_SECRET_FILE` accordingly.
- Top-level `secrets:` block declares `gitea_cache_secret` as `external: true`, with the create command commented inline next to it.
- `poc/gitea/config.yaml`: point `cache.{host,port,external_server}` at `gitea_cache_server:37051`; `external_secret` left empty because the secret is loaded from the file.
## Verification
- Both YAMLs parse (PyYAML).
- Service name `gitea_cache_server` resolves on the `gitea` overlay via Swarm service-name DNS.
- `docker secret inspect gitea_cache_secret` works after the one-time create step.
## Deploy notes
1. On the swarm manager, create the secret once:
```
echo -n "$(openssl rand -hex 32)" | docker secret create gitea_cache_secret -
```
2. `docker stack deploy -c stack-compose.yml gitea`.
3. Re-run a CI job and confirm cache `restore`/`save` log lines no longer ETIMEDOUT.
## Related
- Related: https://gitea.rudolf.keenetic.link/admin/brencher/pulls/15 — `python-package.yml` enables `cache: 'pip'` and was timing out for the reason this PR fixes.
- Refs: gitea/actions/runs/16 jobs #248/249/250.
## Supersedes
- Supersedes: admin/ansible#33 — first attempt was based on a stale `origin/master` and accidentally included `hermes_squid/` changes from the actual upstream merge. The current branch is rebased on `admin/master` (`c3b7128`) and only touches `poc/gitea/`.
The built-in act_runner cache-server binds to an interface unreachable
from per-job container networks (default container.network: ""), so
ACTIONS_CACHE_URL resolves to 192.168.112.21:37051 and every job burns
~9 minutes on:
::warning::Failed to save: reserveCache failed: connect ETIMEDOUT
::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT
Measured on PR #15 in admin/brencher (gitea/actions/runs/16):
each of 3 jobs ~12 min wall-clock, of which 9 min is dead air waiting
for cache timeouts.
Run a dedicated act_runner cache-server on the same overlay network and
point the runner at it via external_server. Shared secret comes from a
Docker secret (gitea_cache_secret) read by both services via
ACT_RUNNER_EXTERNAL_SECRET_FILE.
The image's /usr/local/bin/run.sh entrypoint ignores argv and unconditionally
runs 'act_runner register && act_runner daemon' (the runner path), so
cache_server.service.command must call the act_runner binary directly with
the 'cache-server' subcommand: command: ["act_runner", "cache-server"].
Related: admin/brencher#15
Refs: gitea/actions/runs/16 (jobs #248/249/250)
Tracking issue created for this fix and the upstream CI improvement it unblocks:
→ #35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup)
Closes nothing (this PR is what #35 tracks), but linked for cross-reference.
Tracking issue created for this fix and the upstream CI improvement it unblocks:
→ https://gitea.rudolf.keenetic.link/admin/ansible/issues/35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup)
Closes nothing (this PR is what #35 tracks), but linked for cross-reference.
admin
merged commit f071c7fb43 into master2026-09-04 08:11:43 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Problem
The built-in
act_runnercache server binds to an interface unreachable from per-job container networks (container.network: ""is the default).ACTIONS_CACHE_URLends up pointing at an unreachable IP, and every CI job burns ~9 minutes on:Measured on
admin/brencher#15(https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16):So even though
python-package.ymlenablescache: 'pip', the cache backend is unreachable, jobs hitpip installcold every time, and CI ends up ~4x slower than necessary.Fix
Run a dedicated
act_runner cache-servercontainer on the same overlay network as the runner, and point the runner at it viaexternal_server. Shared secret comes from a Docker secret mounted into both containers and read viaACT_RUNNER_EXTERNAL_SECRET_FILE.Changes
poc/gitea/stack-compose.yml:cache_serverservice (command: ["cache-server"]) on thegiteaoverlay with acache_datanamed volume.runnerandcache_serverboth mount the external Docker secretgitea_cache_secretat/run/secrets/gitea_cache_secretand setACT_RUNNER_EXTERNAL_SECRET_FILEaccordingly.secrets:block declaresgitea_cache_secretasexternal: true, with the create command commented inline next to it.poc/gitea/config.yaml: pointcache.{host,port,external_server}atgitea_cache_server:37051;external_secretleft empty because the secret is loaded from the file.Verification
gitea_cache_serverresolves on thegiteaoverlay via Swarm service-name DNS.docker secret inspect gitea_cache_secretworks after the one-time create step.Deploy notes
docker stack deploy -c stack-compose.yml gitea.restore/savelog lines no longer ETIMEDOUT.Related
python-package.ymlenablescache: 'pip'and was timing out for the reason this PR fixes.Supersedes
origin/masterand accidentally includedhermes_squid/changes from the actual upstream merge. The current branch is rebased onadmin/master(c3b7128) and only touchespoc/gitea/.b02fc5ecb8toee84ae59cdee84ae59cdto7b74e58bff7b74e58bffto0d4a1628290d4a162829to4775ed0cc84775ed0cc8to1ec79690091ec7969009to2b0ac25a682b0ac25a68tob6f699617cb6f699617cto36de23e858The built-in act_runner cache-server binds to an interface unreachable from per-job container networks (default container.network: ""), so ACTIONS_CACHE_URL resolves to 192.168.112.21:37051 and every job burns ~9 minutes on: ::warning::Failed to save: reserveCache failed: connect ETIMEDOUT ::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT Measured on PR #15 in admin/brencher (gitea/actions/runs/16): each of 3 jobs ~12 min wall-clock, of which 9 min is dead air waiting for cache timeouts. Run a dedicated act_runner cache-server on the same overlay network and point the runner at it via external_server. Shared secret comes from a Docker secret (gitea_cache_secret) read by both services via ACT_RUNNER_EXTERNAL_SECRET_FILE. The image's /usr/local/bin/run.sh entrypoint ignores argv and unconditionally runs 'act_runner register && act_runner daemon' (the runner path), so cache_server.service.command must call the act_runner binary directly with the 'cache-server' subcommand: command: ["act_runner", "cache-server"]. Related: admin/brencher#15 Refs: gitea/actions/runs/16 (jobs #248/249/250)36de23e858to1ba50f8e60Tracking issue created for this fix and the upstream CI improvement it unblocks:
→ #35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup)
Closes nothing (this PR is what #35 tracks), but linked for cross-reference.