Optimizes .github/workflows/python-package.yml so the build job runs in ~3-4 minutes instead of ~13 minutes (which is what the recent run on PR #5 took — see https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/14, started 23:52, completed 00:05, 13m 19s wall-clock).
Why it was slow
The current matrix has 3 Python versions × 5 checkers = 15 separate CI jobs, each one:
Checking out the repo (~10s)
Setting up Python via actions/setup-python@v3 with no cache enabled (~15-30s)
Running pip install -r requirements.txt -r requirements-dev.txtfrom scratch every single time (~30-90s per job × 15 jobs = 7-22 minutes wasted)
The two debug entries (python -c 'import sys; print(...)' and pip list) were not real checks — they were noise from the upstream github build-and-test-python template, left in place but never reviewed.
What's in this PR
.github/workflows/python-package.yml — single file, +41/-56
cache: 'pip' enabled with cache-dependency-path: requirements.txt, requirements-dev.txt — first run downloads and caches, subsequent runs hit the cache. Drops pip install from ~60s to ~5-15s when requirements don't change.
Two debug-only matrix entries removed (python -c ... and pip list) — they were not real checks.
Matrix restructured: one Python-version axis only, no checker axis. Each Python job runs install + flake8 + mypy sequentially. 15 jobs → 3 jobs (~5x fewer fresh venvs to set up).
concurrency: cancel-in-progress: true added — a new push to the same PR auto-cancels the previous in-flight run, saving runner minutes when iterating on a fix.
Bumped actions/setup-python from v3 to v5 (required for cache: 'pip').
setup-python@v5 with cache: 'pip' + cache-dependency-path pointing at requirements.txt and requirements-dev.txt
concurrency block correctly scoped to ci-${{ github.workflow }}-${{ github.ref }}
Expected timing
Scenario
Before
After
First run (cold cache)
~13m 19s
~5-7m
Subsequent runs (warm cache, no req changes)
~13m 19s
~3-4m
Re-run after requirements.txt change
~13m 19s
~4-5m
What was NOT verified
Actual Gitea Actions runner behaviour with cache: 'pip' — the Gitea Actions runner is GitHub Actions-compatible, and actions/setup-python@v5 with cache: 'pip' is a well-known idiom there, but the local Gitea runner config may or may not honour it. If it doesn't, the 15→3 fan-out reduction alone is still a ~5x speedup.
PR #14 ("re-enable the tests job") is still open. If/when it merges, this PR will need a follow-up to re-add the tests: job with the same cache: 'pip' setup applied — otherwise the playwright install will dominate again.
Commit history
* db3807e (HEAD -> feat/optimize-ci-build-time, origin/feat/optimize-ci-build-time)
ci: speed up python-package.yml from ~13min to ~3-4min
* 70f4a41 ci: comment out the tests job in python-package.yml (#13) (upstream/main tip)
## Summary
Optimizes `.github/workflows/python-package.yml` so the build job runs in ~3-4 minutes instead of ~13 minutes (which is what the recent run on PR #5 took — see https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/14, started 23:52, completed 00:05, **13m 19s wall-clock**).
## Why it was slow
The current matrix has **3 Python versions × 5 checkers = 15 separate CI jobs**, each one:
1. Checking out the repo (~10s)
2. Setting up Python via `actions/setup-python@v3` with **no cache enabled** (~15-30s)
3. Running `pip install -r requirements.txt -r requirements-dev.txt` **from scratch every single time** (~30-90s per job × 15 jobs = 7-22 minutes wasted)
The two debug entries (`python -c 'import sys; print(...)'` and `pip list`) were not real checks — they were noise from the upstream github `build-and-test-python` template, left in place but never reviewed.
## What's in this PR
- `.github/workflows/python-package.yml` — single file, +41/-56
1. **`cache: 'pip'` enabled** with `cache-dependency-path: requirements.txt, requirements-dev.txt` — first run downloads and caches, subsequent runs hit the cache. Drops pip install from ~60s to ~5-15s when requirements don't change.
2. **Two debug-only matrix entries removed** (`python -c ...` and `pip list`) — they were not real checks.
3. **Matrix restructured**: one Python-version axis only, no checker axis. Each Python job runs `install + flake8 + mypy` sequentially. **15 jobs → 3 jobs** (~5x fewer fresh venvs to set up).
4. **`concurrency: cancel-in-progress: true`** added — a new push to the same PR auto-cancels the previous in-flight run, saving runner minutes when iterating on a fix.
5. Bumped `actions/setup-python` from `v3` to `v5` (required for `cache: 'pip'`).
## Verification
- YAML parses via PyYAML — ok
- `jobs` dict contains `['build']` only
- `build.strategy.matrix.python-version` = `['3.12.3', '3.13', '3.14']`
- **total CI jobs: 3** (was 15)
- `setup-python@v5` with `cache: 'pip'` + `cache-dependency-path` pointing at `requirements.txt` and `requirements-dev.txt`
- `concurrency` block correctly scoped to `ci-${{ github.workflow }}-${{ github.ref }}`
## Expected timing
| Scenario | Before | After |
|---|---|---|
| First run (cold cache) | ~13m 19s | ~5-7m |
| Subsequent runs (warm cache, no req changes) | ~13m 19s | ~3-4m |
| Re-run after `requirements.txt` change | ~13m 19s | ~4-5m |
## What was NOT verified
- Actual Gitea Actions runner behaviour with `cache: 'pip'` — the Gitea Actions runner is GitHub Actions-compatible, and `actions/setup-python@v5` with `cache: 'pip'` is a well-known idiom there, but the local Gitea runner config may or may not honour it. If it doesn't, the 15→3 fan-out reduction alone is still a ~5x speedup.
## Note on PR #14
PR #14 ("re-enable the tests job") is still open. If/when it merges, this PR will need a follow-up to re-add the `tests:` job with the same `cache: 'pip'` setup applied — otherwise the playwright install will dominate again.
## Commit history
```
* db3807e (HEAD -> feat/optimize-ci-build-time, origin/feat/optimize-ci-build-time)
ci: speed up python-package.yml from ~13min to ~3-4min
* 70f4a41 ci: comment out the tests job in python-package.yml (#13) (upstream/main tip)
```
The build job was running for ~13 minutes per PR even with tests
disabled (per PR #13). The actual cause: the matrix had
3 python-versions x 5 checkers = 15 separate CI jobs, each
re-installing the same requirements.txt + requirements-dev.txt
from scratch with no pip cache.
This commit:
1. Enables pip caching via actions/setup-python@v5 cache: 'pip'
keyed on the contents of requirements.txt and requirements-dev.txt.
When those files don't change, pip install is near-instant.
When they do change, only the diff downloads.
2. Drops two debug-only matrix entries that were never real checks:
python -c 'import sys; print(...)' (debug)
pip list (debug)
They took up two of the five matrix cells but produced no
actionable signal.
3. Restructures the matrix so each Python version runs ONE job
that does install + flake8 (2 invocations) + mypy sequentially,
instead of one job per (version, checker) pair. Net: 3 jobs
total instead of 15.
4. Adds a concurrency: cancel-in-progress block so a new push
to the same PR auto-cancels the previous in-flight run,
saving runner minutes when iterating on a fix.
Verified:
- YAML parses via PyYAML
- jobs dict contains ['build'] only
- matrix python versions: ['3.12.3', '3.13', '3.14']
- total CI jobs: 3 (was 15)
- actions/setup-python@v5 with cache: 'pip' + cache-dependency-path
on requirements.txt and requirements-dev.txt
Expected wall-clock (rough estimate, depends on cache hits):
Before: ~13 min (15 jobs, no pip cache, install on every job)
After (cold): ~5-7 min (3 jobs, pip cache miss on first run)
After (warm): ~3-4 min (3 jobs, pip cache hits)
If caching doesn't activate (e.g. self-hosted runner ignores cache:),
the win comes mostly from 15 -> 3 jobs (~5x faster).
Note: PR #14 (re-enable tests) is still open. If/when it merges, this
PR will need a follow-up to re-add the tests job with the same
cache setup applied.
Identified the underlying reason this PR's cache: 'pip' did not actually cache anything: the built-in act_runner cache server was unreachable from per-job container networks (container.network: ""), causing ACTIONS_CACHE_URL=192.168.112.21:37051 to ETIMEDOUT for ~9 min/job (verified in run #16 jobs #248/249/250).
Fix in infra:
admin/ansible#33 — fix(hermes_infra/gitea): add dedicated act_runner cache-server. Adds a gitea_cache_server service to poc/gitea/stack-compose.yml, points cache.{host,port,external_server} at it in poc/gitea/config.yaml, and shares a bearer secret between runner and cache-server via ACT_RUNNER_EXTERNAL_SECRET.
Once that PR is merged and deployed, cache: 'pip' in this workflow will start hitting the cache and the wall-clock per job should drop from ~12 min to the ~3-4 min the PR description promises.
Identified the underlying reason this PR's `cache: 'pip'` did not actually cache anything: the built-in `act_runner` cache server was unreachable from per-job container networks (`container.network: ""`), causing `ACTIONS_CACHE_URL=192.168.112.21:37051` to ETIMEDOUT for ~9 min/job (verified in run #16 jobs #248/249/250).
Fix in infra:
- **https://gitea.rudolf.keenetic.link/admin/ansible/pulls/33** — `fix(hermes_infra/gitea): add dedicated act_runner cache-server`. Adds a `gitea_cache_server` service to `poc/gitea/stack-compose.yml`, points `cache.{host,port,external_server}` at it in `poc/gitea/config.yaml`, and shares a bearer secret between runner and cache-server via `ACT_RUNNER_EXTERNAL_SECRET`.
Once that PR is merged and deployed, `cache: 'pip'` in this workflow will start hitting the cache and the wall-clock per job should drop from ~12 min to the ~3-4 min the PR description promises.
Opened admin/ansible#34 (supersedes #33) with the actual fix for the cache timeout — the first attempt accidentally pulled in hermes_squid/ changes from upstream. The new PR touches only poc/gitea/:
poc/gitea/stack-compose.yml: new cache_server service on the gitea overlay; shared secret mounted from Docker secret gitea_cache_secret (no more CHANGE_ME_SHARED_SECRET placeholder in env).
poc/gitea/config.yaml: cache.{host,port,external_server} point at gitea_cache_server:37051; external_secret empty because the secret is loaded from the file by ACT_RUNNER_EXTERNAL_SECRET_FILE.
Diff: +32/-19 across 2 files. Once #34 is merged and deployed, cache: 'pip' here should start hitting the cache.
Opened **admin/ansible#34** (supersedes #33) with the actual fix for the cache timeout — the first attempt accidentally pulled in `hermes_squid/` changes from upstream. The new PR touches only `poc/gitea/`:
- `poc/gitea/stack-compose.yml`: new `cache_server` service on the `gitea` overlay; shared secret mounted from Docker secret `gitea_cache_secret` (no more `CHANGE_ME_SHARED_SECRET` placeholder in env).
- `poc/gitea/config.yaml`: `cache.{host,port,external_server}` point at `gitea_cache_server:37051`; `external_secret` empty because the secret is loaded from the file by `ACT_RUNNER_EXTERNAL_SECRET_FILE`.
Diff: +32/-19 across 2 files. Once #34 is merged and deployed, `cache: 'pip'` here should start hitting the cache.
→ https://gitea.rudolf.keenetic.link/admin/ansible/pulls/34
Speedup: ~5.2× per run (35m → 7m). Per job the wall-clock dropped from ~11-12 min to ~2-2.5 min, which matches PR #15's prediction (3-4 min) — we hit the optimistic end of the band because pip cache is fully warm.
Infra fix: admin/ansible#34 — dedicated gitea_cache_server service on the gitea overlay, shared secret via Docker secret gitea_cache_secret (ACT_RUNNER_EXTERNAL_SECRET_FILE), entrypoint scripts that substitute the secret into a runtime copy of config.yaml's cache.external_secret.
**Verification ✅** — admin/ansible#34 deploy completed; cache is reachable and the CI pipeline is ~5x faster.
| Run | Job | wall-clock |
|-----|-----|------------|
| **#16** (before fix, cache ETIMEDOUT ~9 min/job) | 3.12.3 | 12m 41s |
| | 3.13 | 11m 13s |
| | 3.14 | 11m 18s |
| | **total** | **35m 12s** |
| **#59** (after fix, cache works) | 3.12.3 | **2m 10s** |
| | 3.13 | **2m 12s** |
| | 3.14 | **2m 22s** |
| | **total** | **6m 44s** |
**Speedup: ~5.2× per run** (35m → 7m). Per job the wall-clock dropped from ~11-12 min to ~2-2.5 min, which matches PR #15's prediction (3-4 min) — we hit the optimistic end of the band because pip cache is fully warm.
Verified runs:
- https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16 (before)
- https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/59 (after)
Infra fix: https://gitea.rudolf.keenetic.link/admin/ansible/pulls/34 — dedicated `gitea_cache_server` service on the `gitea` overlay, shared secret via Docker secret `gitea_cache_secret` (`ACT_RUNNER_EXTERNAL_SECRET_FILE`), entrypoint scripts that substitute the secret into a runtime copy of config.yaml's `cache.external_secret`.
Tracking issue created for this fix and the upstream CI improvement it unblocks:
→ admin/ansible#35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup)
Closes nothing (this PR is what #35 tracks), but linked for cross-reference.
Tracking issue created for this fix and the upstream CI improvement it unblocks:
→ https://gitea.rudolf.keenetic.link/admin/ansible/issues/35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup)
Closes nothing (this PR is what #35 tracks), but linked for cross-reference.
Drop the CI-history and per-option narration comments; keep only what is
needed for the workflow to be readable (job names, step names). Mirrors
the repo's compose/env/entrypoint-with-no-explanatory-comments style.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Optimizes
.github/workflows/python-package.ymlso the build job runs in ~3-4 minutes instead of ~13 minutes (which is what the recent run on PR #5 took — see https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/14, started 23:52, completed 00:05, 13m 19s wall-clock).Why it was slow
The current matrix has 3 Python versions × 5 checkers = 15 separate CI jobs, each one:
actions/setup-python@v3with no cache enabled (~15-30s)pip install -r requirements.txt -r requirements-dev.txtfrom scratch every single time (~30-90s per job × 15 jobs = 7-22 minutes wasted)The two debug entries (
python -c 'import sys; print(...)'andpip list) were not real checks — they were noise from the upstream githubbuild-and-test-pythontemplate, left in place but never reviewed.What's in this PR
.github/workflows/python-package.yml— single file, +41/-56cache: 'pip'enabled withcache-dependency-path: requirements.txt, requirements-dev.txt— first run downloads and caches, subsequent runs hit the cache. Drops pip install from ~60s to ~5-15s when requirements don't change.python -c ...andpip list) — they were not real checks.install + flake8 + mypysequentially. 15 jobs → 3 jobs (~5x fewer fresh venvs to set up).concurrency: cancel-in-progress: trueadded — a new push to the same PR auto-cancels the previous in-flight run, saving runner minutes when iterating on a fix.actions/setup-pythonfromv3tov5(required forcache: 'pip').Verification
jobsdict contains['build']onlybuild.strategy.matrix.python-version=['3.12.3', '3.13', '3.14']setup-python@v5withcache: 'pip'+cache-dependency-pathpointing atrequirements.txtandrequirements-dev.txtconcurrencyblock correctly scoped toci-${{ github.workflow }}-${{ github.ref }}Expected timing
requirements.txtchangeWhat was NOT verified
cache: 'pip'— the Gitea Actions runner is GitHub Actions-compatible, andactions/setup-python@v5withcache: 'pip'is a well-known idiom there, but the local Gitea runner config may or may not honour it. If it doesn't, the 15→3 fan-out reduction alone is still a ~5x speedup.Note on PR #14
PR #14 ("re-enable the tests job") is still open. If/when it merges, this PR will need a follow-up to re-add the
tests:job with the samecache: 'pip'setup applied — otherwise the playwright install will dominate again.Commit history
Identified the underlying reason this PR's
cache: 'pip'did not actually cache anything: the built-inact_runnercache server was unreachable from per-job container networks (container.network: ""), causingACTIONS_CACHE_URL=192.168.112.21:37051to ETIMEDOUT for ~9 min/job (verified in run #16 jobs #248/249/250).Fix in infra:
fix(hermes_infra/gitea): add dedicated act_runner cache-server. Adds agitea_cache_serverservice topoc/gitea/stack-compose.yml, pointscache.{host,port,external_server}at it inpoc/gitea/config.yaml, and shares a bearer secret between runner and cache-server viaACT_RUNNER_EXTERNAL_SECRET.Once that PR is merged and deployed,
cache: 'pip'in this workflow will start hitting the cache and the wall-clock per job should drop from ~12 min to the ~3-4 min the PR description promises.Opened admin/ansible#34 (supersedes #33) with the actual fix for the cache timeout — the first attempt accidentally pulled in
hermes_squid/changes from upstream. The new PR touches onlypoc/gitea/:poc/gitea/stack-compose.yml: newcache_serverservice on thegiteaoverlay; shared secret mounted from Docker secretgitea_cache_secret(no moreCHANGE_ME_SHARED_SECRETplaceholder in env).poc/gitea/config.yaml:cache.{host,port,external_server}point atgitea_cache_server:37051;external_secretempty because the secret is loaded from the file byACT_RUNNER_EXTERNAL_SECRET_FILE.Diff: +32/-19 across 2 files. Once #34 is merged and deployed,
cache: 'pip'here should start hitting the cache.→ admin/ansible#34
Verification ✅ — admin/ansible#34 deploy completed; cache is reachable and the CI pipeline is ~5x faster.
Speedup: ~5.2× per run (35m → 7m). Per job the wall-clock dropped from ~11-12 min to ~2-2.5 min, which matches PR #15's prediction (3-4 min) — we hit the optimistic end of the band because pip cache is fully warm.
Verified runs:
Infra fix: admin/ansible#34 — dedicated
gitea_cache_serverservice on thegiteaoverlay, shared secret via Docker secretgitea_cache_secret(ACT_RUNNER_EXTERNAL_SECRET_FILE), entrypoint scripts that substitute the secret into a runtime copy of config.yaml'scache.external_secret.Tracking issue created for this fix and the upstream CI improvement it unblocks:
→ admin/ansible#35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup)
Closes nothing (this PR is what #35 tracks), but linked for cross-reference.