fix(hermes_infra/gitea): add dedicated act_runner cache-server (#34)
## Problem The built-in `act_runner` cache server binds to an interface unreachable from per-job container networks (`container.network: ""` is the default). `ACTIONS_CACHE_URL` ends up pointing at an unreachable IP, and every CI job burns ~9 minutes on: ``` ::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051 ::warning::Failed to save: reserveCache failed: connect ETIMEDOUT 192.168.112.21:37051 ``` Measured on `admin/brencher#15` (https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16): | Job | wall-clock | of which dead air on cache | |---------|------------|----------------------------| | 3.12.3 | 12m 40s | ~9m (271s × 2 timeouts) | | 3.13 | 11m 13s | ~9m | | 3.14 | 11m 17s | ~9m | So even though `python-package.yml` enables `cache: 'pip'`, the cache backend is unreachable, jobs hit `pip install` cold every time, and CI ends up ~4x slower than necessary. ## Fix Run a dedicated `act_runner cache-server` container on the same overlay network as the runner, and point the runner at it via `external_server`. Shared secret comes from a Docker secret mounted into both containers and read via `ACT_RUNNER_EXTERNAL_SECRET_FILE`. ## Changes - `poc/gitea/stack-compose.yml`: - New `cache_server` service (`command: ["cache-server"]`) on the `gitea` overlay with a `cache_data` named volume. - `runner` and `cache_server` both mount the external Docker secret `gitea_cache_secret` at `/run/secrets/gitea_cache_secret` and set `ACT_RUNNER_EXTERNAL_SECRET_FILE` accordingly. - Top-level `secrets:` block declares `gitea_cache_secret` as `external: true`, with the create command commented inline next to it. - `poc/gitea/config.yaml`: point `cache.{host,port,external_server}` at `gitea_cache_server:37051`; `external_secret` left empty because the secret is loaded from the file. ## Verification - Both YAMLs parse (PyYAML). - Service name `gitea_cache_server` resolves on the `gitea` overlay via Swarm service-name DNS. - `docker secret inspect gitea_cache_secret` works after the one-time create step. ## Deploy notes 1. On the swarm manager, create the secret once: ``` echo -n "$(openssl rand -hex 32)" | docker secret create gitea_cache_secret - ``` 2. `docker stack deploy -c stack-compose.yml gitea`. 3. Re-run a CI job and confirm cache `restore`/`save` log lines no longer ETIMEDOUT. ## Related - Related: admin/brencher#15 — `python-package.yml` enables `cache: 'pip'` and was timing out for the reason this PR fixes. - Refs: gitea/actions/runs/16 jobs #248/249/250. ## Supersedes - Supersedes: admin/ansible#33 — first attempt was based on a stale `origin/master` and accidentally included `hermes_squid/` changes from the actual upstream merge. The current branch is rebased on `admin/master` (`c3b7128`) and only touches `poc/gitea/`.Reviewed-on: http://192.168.1.96/admin/ansible/pulls/34 Reviewed-by: admin <rudolfss13@gmail.com> Co-authored-by: hermes <rudolfss13+hermes@gmail.com>
This commit was merged in pull request #34.
This commit is contained in:
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Custom entrypoint for the act_runner cache-server.
|
||||
#
|
||||
# The upstream image's /usr/local/bin/run.sh hardcodes `act_runner daemon`
|
||||
# and ignores argv. So we replace the entrypoint here.
|
||||
#
|
||||
# cache-server also requires cache.external_secret to be set in config.yaml
|
||||
# as a literal value (not a file path). We keep the value in a Docker
|
||||
# secret and substitute it into a runtime copy of config.yaml before
|
||||
# exec'ing act_runner.
|
||||
set -eu
|
||||
|
||||
SECRET_FILE="${ACT_RUNNER_EXTERNAL_SECRET_FILE:-/run/secrets/gitea_cache_secret}"
|
||||
CONFIG_FILE="${CONFIG_FILE:-/cfg/config.yaml}"
|
||||
|
||||
if [[ ! -s "$SECRET_FILE" ]]; then
|
||||
echo "cache-server-run: secret file $SECRET_FILE is missing or empty" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -f "$CONFIG_FILE" ]]; then
|
||||
echo "cache-server-run: config file $CONFIG_FILE not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SECRET_VALUE="$(cat "$SECRET_FILE")"
|
||||
|
||||
# Render a copy of config.yaml with cache.external_secret replaced by the
|
||||
# secret's literal value. /cfg/config.yaml is a read-only Swarm config mount,
|
||||
# so we cannot edit it in place. Use a plain-file replacement — the
|
||||
# upstream config.yaml has the form `external_secret: ""` (or any literal)
|
||||
# on a single line, so a simple sed is safe.
|
||||
RUNTIME_CONFIG="/tmp/config.runtime.yaml"
|
||||
cp "$CONFIG_FILE" "$RUNTIME_CONFIG"
|
||||
# Replace the value after `external_secret:` on a single line.
|
||||
# Use a delimiter unlikely to appear in the secret.
|
||||
sed -i "s|^ external_secret:.*$| external_secret: \"${SECRET_VALUE}\"|" "$RUNTIME_CONFIG"
|
||||
|
||||
exec /usr/local/bin/act_runner cache-server --config "$RUNTIME_CONFIG"
|
||||
+8
-18
@@ -70,26 +70,16 @@ runner:
|
||||
- "ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04"
|
||||
|
||||
cache:
|
||||
# Enable cache server to use actions/cache.
|
||||
enabled: true
|
||||
# The directory to store the cache data.
|
||||
# If it's empty, the cache data will be stored in $HOME/.cache/actcache.
|
||||
dir: ""
|
||||
# The host of the cache server.
|
||||
# It's not for the address to listen, but the address to connect from job containers.
|
||||
# So 0.0.0.0 is a bad choice, leave it empty to detect automatically.
|
||||
host: ""
|
||||
# The port of the cache server.
|
||||
# 0 means to use a random available port.
|
||||
port: 0
|
||||
# The external cache server URL. Valid only when enable is true.
|
||||
# If it's specified, act_runner will use this URL as the ACTIONS_CACHE_URL rather than start a server by itself.
|
||||
# The URL should generally end with "/".
|
||||
# Requires external_secret below to be set to the same value on both this runner and the cache-server.
|
||||
external_server: ""
|
||||
# Shared secret between this runner and the external `act_runner cache-server`. Required when external_server
|
||||
# (or `act_runner cache-server`) is in use: the runner pre-registers each job's ACTIONS_RUNTIME_TOKEN with the
|
||||
# cache-server, and the cache-server enforces bearer auth + per-repo cache isolation.
|
||||
# Point at the dedicated gitea_cache_server container on the `gitea` overlay.
|
||||
# The runner's built-in cache server binds to an interface unreachable from
|
||||
# per-job container networks, causing ~9 min/job wasted on
|
||||
# "reserveCache failed: connect ETIMEDOUT" (see gitea/actions/runs/16 job #248).
|
||||
host: "gitea_cache_server"
|
||||
port: 37051
|
||||
external_server: "http://gitea_cache_server:37051/"
|
||||
# Read from the gitea_cache_secret Docker secret (see stack-compose.yml).
|
||||
external_secret: ""
|
||||
|
||||
container:
|
||||
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Custom entrypoint for the gitea_runner service.
|
||||
#
|
||||
# Same trick as cache-server-run.sh: act_runner requires cache.external_secret
|
||||
# to be a literal string in config.yaml, not a file path. We keep the value in
|
||||
# a Docker secret and substitute it into a runtime copy of config.yaml, then
|
||||
# exec the image's own /usr/local/bin/run.sh (which handles registration +
|
||||
# daemon).
|
||||
set -eu
|
||||
|
||||
SECRET_FILE="${ACT_RUNNER_EXTERNAL_SECRET_FILE:-/run/secrets/gitea_cache_secret}"
|
||||
CONFIG_FILE="${CONFIG_FILE:-/cfg/config.yaml}"
|
||||
|
||||
if [[ ! -s "$SECRET_FILE" ]]; then
|
||||
echo "runner-run: secret file $SECRET_FILE is missing or empty" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -f "$CONFIG_FILE" ]]; then
|
||||
echo "runner-run: config file $CONFIG_FILE not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SECRET_VALUE="$(cat "$SECRET_FILE")"
|
||||
|
||||
# Render a runtime copy with the secret substituted in.
|
||||
RUNTIME_CONFIG="/tmp/config.runtime.yaml"
|
||||
cp "$CONFIG_FILE" "$RUNTIME_CONFIG"
|
||||
sed -i "s|^ external_secret:.*$| external_secret: \"${SECRET_VALUE}\"|" "$RUNTIME_CONFIG"
|
||||
|
||||
# The upstream run.sh reads $CONFIG_FILE from the environment to pass
|
||||
# --config to act_runner. Override it so registration + daemon both pick up
|
||||
# the runtime copy with the substituted secret.
|
||||
export CONFIG_FILE="$RUNTIME_CONFIG"
|
||||
|
||||
exec /usr/local/bin/run.sh
|
||||
@@ -1,4 +1,8 @@
|
||||
|
||||
# Gitea Actions stack. Uses a dedicated act_runner cache-server so that
|
||||
# job-container networks can reach the cache backend via Swarm service DNS
|
||||
# (gitea_cache_server:37051) instead of the unreachable IP the built-in
|
||||
# cache server binds to per-job.
|
||||
services:
|
||||
server:
|
||||
image: docker.gitea.com/gitea:1.27.0
|
||||
@@ -19,6 +23,7 @@ services:
|
||||
|
||||
runner:
|
||||
image: docker.io/gitea/act_runner:nightly
|
||||
entrypoint: ["/usr/local/bin/runner-run.sh"]
|
||||
environment:
|
||||
CONFIG_FILE: /cfg/config.yaml
|
||||
GITEA_INSTANCE_URL: "https://gitea.rudolf.keenetic.link"
|
||||
@@ -26,14 +31,51 @@ services:
|
||||
GITEA_RUNNER_NAME: "runner1"
|
||||
GITEA_RUNNER_LABELS: "runner_label1"
|
||||
# GITEA_RUNNER_EPHEMERAL: "1"
|
||||
ACT_RUNNER_EXTERNAL_SECRET_FILE: /run/secrets/gitea_cache_secret
|
||||
networks:
|
||||
- gitea
|
||||
# - nginx_ingress
|
||||
configs:
|
||||
- source: runner_config
|
||||
target: /cfg/config.yaml
|
||||
mode: 0444
|
||||
- source: runner_run
|
||||
target: /usr/local/bin/runner-run.sh
|
||||
mode: 0755
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
secrets:
|
||||
- gitea_cache_secret
|
||||
deploy:
|
||||
restart_policy:
|
||||
condition: on-failure
|
||||
delay: 5s
|
||||
max_attempts: 3
|
||||
|
||||
cache_server:
|
||||
# The image's /usr/local/bin/run.sh hardcodes `act_runner daemon` and
|
||||
# ignores argv, so passing `cache-server` via `command` doesn't help —
|
||||
# the daemon path is always taken. We replace the entrypoint by
|
||||
# mounting our own script via a Swarm config (file mount), then have it
|
||||
# substitute the Docker secret literal into a runtime copy of
|
||||
# config.yaml's cache.external_secret and exec act_runner.
|
||||
image: docker.io/gitea/act_runner:nightly
|
||||
entrypoint: ["/usr/local/bin/cache-server-run.sh"]
|
||||
environment:
|
||||
ACT_RUNNER_EXTERNAL_SECRET_FILE: /run/secrets/gitea_cache_secret
|
||||
networks:
|
||||
- gitea
|
||||
volumes:
|
||||
- cache_data:/root/.cache/actcache
|
||||
secrets:
|
||||
- gitea_cache_secret
|
||||
configs:
|
||||
- source: runner_config
|
||||
target: /cfg/config.yaml
|
||||
mode: 0444
|
||||
- source: cache_server_run
|
||||
target: /usr/local/bin/cache-server-run.sh
|
||||
mode: 0755
|
||||
|
||||
networks:
|
||||
nginx_ingress:
|
||||
@@ -44,6 +86,17 @@ networks:
|
||||
configs:
|
||||
runner_config:
|
||||
file: ./config.yaml
|
||||
cache_server_run:
|
||||
file: ./cache-server-run.sh
|
||||
runner_run:
|
||||
file: ./runner-run.sh
|
||||
|
||||
secrets:
|
||||
gitea_cache_secret:
|
||||
# Create once before first deploy (or to rotate):
|
||||
# echo -n "$SECRET" | docker secret create gitea_cache_secret -
|
||||
# Then `docker stack deploy -c stack-compose.yml gitea`.
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
data:
|
||||
@@ -52,4 +105,5 @@ volumes:
|
||||
type: nfs
|
||||
o: addr=192.168.1.96
|
||||
device: :/root/volumes/gitea
|
||||
# device: :/media/DATA/runtime/gitea
|
||||
# device: :/media/DATA/runtime/gitea
|
||||
cache_data:
|
||||
Reference in New Issue
Block a user