fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup) #35

Open
opened 2026-09-04 00:07:29 +02:00 by hermes · 0 comments
Contributor

Summary

Fix CI cache backend for brencher / Gitea Actions so that cache: 'pip' and other workflow cache operations actually work — currently every job wastes ~9 minutes on cache timeouts.

Problem

The gitea/act_runner:nightly image's built-in cache server binds to an interface unreachable from per-job container networks (container.network: "" is the default in the runner config). When a workflow uses cache: 'pip', every actions/setup-python@v5 step performs reserveCache and getCacheEntry HTTP calls against ACTIONS_CACHE_URL=http://192.168.112.21:37051 — and that address is unreachable from the job container, producing two ETIMEDOUT warnings per job:

::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051
::warning::Failed to save:   reserveCache  failed: connect ETIMEDOUT 192.168.112.21:37051

Each timeout burns ~271 s wall-clock. Three matrix jobs × ~9 min of dead air = most of the run.

Fix

Two-piece fix:

  1. Dedicated gitea_cache_server service on the same overlay network as the runner (so per-job containers can reach it via Swarm service DNS gitea_cache_server:37051), configured in poc/gitea/stack-compose.yml.
  2. Shared secret via Docker secret gitea_cache_secret mounted into both runner and cache_server, with the literal value substituted into cache.external_secret in a runtime copy of config.yaml by custom entrypoint scripts (poc/gitea/cache-server-run.sh, poc/gitea/runner-run.sh).

compose-go for Swarm merges entrypoint + command into ContainerSpec.Args, leaving ContainerSpec.Command unset — so command: ["act_runner", "cache-server"] does NOT replace the image ENTRYPOINT (/sbin/tini -- run.sh). The fix replaces the entrypoint script via a Swarm config mount at /usr/local/bin/cache-server-run.sh.

Verification

Run Job wall-clock
#16 (before fix) 3.12.3 12m 41s
3.13 11m 13s
3.14 11m 18s
total 35m 12s
#59 (after fix) 3.12.3 2m 10s
3.13 2m 12s
3.14 2m 22s
total 6m 44s

Speedup: ~5.2× per run, jobs drop to ~2-2.5 min each (matches the optimistic end of PR #15's predicted 3-4 min because pip cache is fully warm).

Linked PRs

  • #34 — the infra fix (gitea_cache_server + shared secret + entrypoint scripts).
  • admin/brencher#15 — enables cache: 'pip' in python-package.yml and shrinks the matrix from 15 jobs to 3.

Deploy notes

The gitea_cache_secret Docker secret must exist before docker stack deploy:

echo -n "$(openssl rand -hex 32)" | docker secret create gitea_cache_secret -

Both services read it via ACT_RUNNER_EXTERNAL_SECRET_FILE=/run/secrets/gitea_cache_secret. Rotate by re-creating the secret and redeploying the stack.

## Summary Fix CI cache backend for brencher / Gitea Actions so that `cache: 'pip'` and other workflow cache operations actually work — currently every job wastes ~9 minutes on cache timeouts. ## Problem The `gitea/act_runner:nightly` image's built-in cache server binds to an interface unreachable from per-job container networks (`container.network: ""` is the default in the runner config). When a workflow uses `cache: 'pip'`, every `actions/setup-python@v5` step performs `reserveCache` and `getCacheEntry` HTTP calls against `ACTIONS_CACHE_URL=http://192.168.112.21:37051` — and that address is unreachable from the job container, producing two ETIMEDOUT warnings per job: ``` ::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051 ::warning::Failed to save: reserveCache failed: connect ETIMEDOUT 192.168.112.21:37051 ``` Each timeout burns ~271 s wall-clock. Three matrix jobs × ~9 min of dead air = most of the run. ## Fix Two-piece fix: 1. **Dedicated `gitea_cache_server` service** on the same overlay network as the runner (so per-job containers can reach it via Swarm service DNS `gitea_cache_server:37051`), configured in `poc/gitea/stack-compose.yml`. 2. **Shared secret** via Docker secret `gitea_cache_secret` mounted into both `runner` and `cache_server`, with the literal value substituted into `cache.external_secret` in a runtime copy of `config.yaml` by custom entrypoint scripts (`poc/gitea/cache-server-run.sh`, `poc/gitea/runner-run.sh`). `compose-go for Swarm` merges `entrypoint` + `command` into `ContainerSpec.Args`, leaving `ContainerSpec.Command` unset — so `command: ["act_runner", "cache-server"]` does NOT replace the image ENTRYPOINT (`/sbin/tini -- run.sh`). The fix replaces the entrypoint script via a Swarm config mount at `/usr/local/bin/cache-server-run.sh`. ## Verification | Run | Job | wall-clock | |-------|--------|------------| | #16 (before fix) | 3.12.3 | 12m 41s | | | 3.13 | 11m 13s | | | 3.14 | 11m 18s | | | **total** | **35m 12s** | | #59 (after fix) | 3.12.3 | **2m 10s** | | | 3.13 | **2m 12s** | | | 3.14 | **2m 22s** | | | **total** | **6m 44s** | **Speedup: ~5.2× per run**, jobs drop to ~2-2.5 min each (matches the optimistic end of PR #15's predicted 3-4 min because pip cache is fully warm). ## Linked PRs - https://gitea.rudolf.keenetic.link/admin/ansible/pulls/34 — the infra fix (gitea_cache_server + shared secret + entrypoint scripts). - https://gitea.rudolf.keenetic.link/admin/brencher/pulls/15 — enables `cache: 'pip'` in `python-package.yml` and shrinks the matrix from 15 jobs to 3. ## Deploy notes The `gitea_cache_secret` Docker secret must exist before `docker stack deploy`: ``` echo -n "$(openssl rand -hex 32)" | docker secret create gitea_cache_secret - ``` Both services read it via `ACT_RUNNER_EXTERNAL_SECRET_FILE=/run/secrets/gitea_cache_secret`. Rotate by re-creating the secret and redeploying the stack.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/ansible#35