ci: speed up python-package.yml from ~13min to ~3-4min #15

Merged
admin merged 7 commits from hermes/brencher:feat/optimize-ci-build-time into main 2026-09-04 09:48:31 +02:00
Contributor

Summary

Optimizes .github/workflows/python-package.yml so the build job runs in ~3-4 minutes instead of ~13 minutes (which is what the recent run on PR #5 took — see https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/14, started 23:52, completed 00:05, 13m 19s wall-clock).

Why it was slow

The current matrix has 3 Python versions × 5 checkers = 15 separate CI jobs, each one:

  1. Checking out the repo (~10s)
  2. Setting up Python via actions/setup-python@v3 with no cache enabled (~15-30s)
  3. Running pip install -r requirements.txt -r requirements-dev.txt from scratch every single time (~30-90s per job × 15 jobs = 7-22 minutes wasted)

The two debug entries (python -c 'import sys; print(...)' and pip list) were not real checks — they were noise from the upstream github build-and-test-python template, left in place but never reviewed.

What's in this PR

  • .github/workflows/python-package.yml — single file, +41/-56
    1. cache: 'pip' enabled with cache-dependency-path: requirements.txt, requirements-dev.txt — first run downloads and caches, subsequent runs hit the cache. Drops pip install from ~60s to ~5-15s when requirements don't change.
    2. Two debug-only matrix entries removed (python -c ... and pip list) — they were not real checks.
    3. Matrix restructured: one Python-version axis only, no checker axis. Each Python job runs install + flake8 + mypy sequentially. 15 jobs → 3 jobs (~5x fewer fresh venvs to set up).
    4. concurrency: cancel-in-progress: true added — a new push to the same PR auto-cancels the previous in-flight run, saving runner minutes when iterating on a fix.
    5. Bumped actions/setup-python from v3 to v5 (required for cache: 'pip').

Verification

  • YAML parses via PyYAML — ok
  • jobs dict contains ['build'] only
  • build.strategy.matrix.python-version = ['3.12.3', '3.13', '3.14']
  • total CI jobs: 3 (was 15)
  • setup-python@v5 with cache: 'pip' + cache-dependency-path pointing at requirements.txt and requirements-dev.txt
  • concurrency block correctly scoped to ci-${{ github.workflow }}-${{ github.ref }}

Expected timing

Scenario Before After
First run (cold cache) ~13m 19s ~5-7m
Subsequent runs (warm cache, no req changes) ~13m 19s ~3-4m
Re-run after requirements.txt change ~13m 19s ~4-5m

What was NOT verified

  • Actual Gitea Actions runner behaviour with cache: 'pip' — the Gitea Actions runner is GitHub Actions-compatible, and actions/setup-python@v5 with cache: 'pip' is a well-known idiom there, but the local Gitea runner config may or may not honour it. If it doesn't, the 15→3 fan-out reduction alone is still a ~5x speedup.

Note on PR #14

PR #14 ("re-enable the tests job") is still open. If/when it merges, this PR will need a follow-up to re-add the tests: job with the same cache: 'pip' setup applied — otherwise the playwright install will dominate again.

Commit history

* db3807e (HEAD -> feat/optimize-ci-build-time, origin/feat/optimize-ci-build-time)
      ci: speed up python-package.yml from ~13min to ~3-4min
* 70f4a41 ci: comment out the tests job in python-package.yml (#13)   (upstream/main tip)
## Summary Optimizes `.github/workflows/python-package.yml` so the build job runs in ~3-4 minutes instead of ~13 minutes (which is what the recent run on PR #5 took — see https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/14, started 23:52, completed 00:05, **13m 19s wall-clock**). ## Why it was slow The current matrix has **3 Python versions × 5 checkers = 15 separate CI jobs**, each one: 1. Checking out the repo (~10s) 2. Setting up Python via `actions/setup-python@v3` with **no cache enabled** (~15-30s) 3. Running `pip install -r requirements.txt -r requirements-dev.txt` **from scratch every single time** (~30-90s per job × 15 jobs = 7-22 minutes wasted) The two debug entries (`python -c 'import sys; print(...)'` and `pip list`) were not real checks — they were noise from the upstream github `build-and-test-python` template, left in place but never reviewed. ## What's in this PR - `.github/workflows/python-package.yml` — single file, +41/-56 1. **`cache: 'pip'` enabled** with `cache-dependency-path: requirements.txt, requirements-dev.txt` — first run downloads and caches, subsequent runs hit the cache. Drops pip install from ~60s to ~5-15s when requirements don't change. 2. **Two debug-only matrix entries removed** (`python -c ...` and `pip list`) — they were not real checks. 3. **Matrix restructured**: one Python-version axis only, no checker axis. Each Python job runs `install + flake8 + mypy` sequentially. **15 jobs → 3 jobs** (~5x fewer fresh venvs to set up). 4. **`concurrency: cancel-in-progress: true`** added — a new push to the same PR auto-cancels the previous in-flight run, saving runner minutes when iterating on a fix. 5. Bumped `actions/setup-python` from `v3` to `v5` (required for `cache: 'pip'`). ## Verification - YAML parses via PyYAML — ok - `jobs` dict contains `['build']` only - `build.strategy.matrix.python-version` = `['3.12.3', '3.13', '3.14']` - **total CI jobs: 3** (was 15) - `setup-python@v5` with `cache: 'pip'` + `cache-dependency-path` pointing at `requirements.txt` and `requirements-dev.txt` - `concurrency` block correctly scoped to `ci-${{ github.workflow }}-${{ github.ref }}` ## Expected timing | Scenario | Before | After | |---|---|---| | First run (cold cache) | ~13m 19s | ~5-7m | | Subsequent runs (warm cache, no req changes) | ~13m 19s | ~3-4m | | Re-run after `requirements.txt` change | ~13m 19s | ~4-5m | ## What was NOT verified - Actual Gitea Actions runner behaviour with `cache: 'pip'` — the Gitea Actions runner is GitHub Actions-compatible, and `actions/setup-python@v5` with `cache: 'pip'` is a well-known idiom there, but the local Gitea runner config may or may not honour it. If it doesn't, the 15→3 fan-out reduction alone is still a ~5x speedup. ## Note on PR #14 PR #14 ("re-enable the tests job") is still open. If/when it merges, this PR will need a follow-up to re-add the `tests:` job with the same `cache: 'pip'` setup applied — otherwise the playwright install will dominate again. ## Commit history ``` * db3807e (HEAD -> feat/optimize-ci-build-time, origin/feat/optimize-ci-build-time) ci: speed up python-package.yml from ~13min to ~3-4min * 70f4a41 ci: comment out the tests job in python-package.yml (#13) (upstream/main tip) ```
hermes added 1 commit 2026-08-26 00:11:23 +02:00
ci: speed up python-package.yml from ~13min to ~3-4min
Python package / build (3.12.3) (pull_request) Successful in 12m41s
Python package / build (3.13) (pull_request) Successful in 11m13s
Python package / build (3.14) (pull_request) Successful in 11m18s
db3807e49a
The build job was running for ~13 minutes per PR even with tests
disabled (per PR #13). The actual cause: the matrix had
3 python-versions x 5 checkers = 15 separate CI jobs, each
re-installing the same requirements.txt + requirements-dev.txt
from scratch with no pip cache.

This commit:
  1. Enables pip caching via actions/setup-python@v5 cache: 'pip'
     keyed on the contents of requirements.txt and requirements-dev.txt.
     When those files don't change, pip install is near-instant.
     When they do change, only the diff downloads.

  2. Drops two debug-only matrix entries that were never real checks:
       python -c 'import sys; print(...)'  (debug)
       pip list                          (debug)
     They took up two of the five matrix cells but produced no
     actionable signal.

  3. Restructures the matrix so each Python version runs ONE job
     that does install + flake8 (2 invocations) + mypy sequentially,
     instead of one job per (version, checker) pair. Net: 3 jobs
     total instead of 15.

  4. Adds a concurrency: cancel-in-progress block so a new push
     to the same PR auto-cancels the previous in-flight run,
     saving runner minutes when iterating on a fix.

Verified:
  - YAML parses via PyYAML
  - jobs dict contains ['build'] only
  - matrix python versions: ['3.12.3', '3.13', '3.14']
  - total CI jobs: 3 (was 15)
  - actions/setup-python@v5 with cache: 'pip' + cache-dependency-path
    on requirements.txt and requirements-dev.txt

Expected wall-clock (rough estimate, depends on cache hits):
  Before: ~13 min (15 jobs, no pip cache, install on every job)
  After (cold): ~5-7 min (3 jobs, pip cache miss on first run)
  After (warm): ~3-4 min (3 jobs, pip cache hits)

If caching doesn't activate (e.g. self-hosted runner ignores cache:),
the win comes mostly from 15 -> 3 jobs (~5x faster).

Note: PR #14 (re-enable tests) is still open. If/when it merges, this
PR will need a follow-up to re-add the tests job with the same
cache setup applied.
Author
Contributor

Identified the underlying reason this PR's cache: 'pip' did not actually cache anything: the built-in act_runner cache server was unreachable from per-job container networks (container.network: ""), causing ACTIONS_CACHE_URL=192.168.112.21:37051 to ETIMEDOUT for ~9 min/job (verified in run #16 jobs #248/249/250).

Fix in infra:

  • admin/ansible#33 — fix(hermes_infra/gitea): add dedicated act_runner cache-server. Adds a gitea_cache_server service to poc/gitea/stack-compose.yml, points cache.{host,port,external_server} at it in poc/gitea/config.yaml, and shares a bearer secret between runner and cache-server via ACT_RUNNER_EXTERNAL_SECRET.

Once that PR is merged and deployed, cache: 'pip' in this workflow will start hitting the cache and the wall-clock per job should drop from ~12 min to the ~3-4 min the PR description promises.

Identified the underlying reason this PR's `cache: 'pip'` did not actually cache anything: the built-in `act_runner` cache server was unreachable from per-job container networks (`container.network: ""`), causing `ACTIONS_CACHE_URL=192.168.112.21:37051` to ETIMEDOUT for ~9 min/job (verified in run #16 jobs #248/249/250). Fix in infra: - **https://gitea.rudolf.keenetic.link/admin/ansible/pulls/33** — `fix(hermes_infra/gitea): add dedicated act_runner cache-server`. Adds a `gitea_cache_server` service to `poc/gitea/stack-compose.yml`, points `cache.{host,port,external_server}` at it in `poc/gitea/config.yaml`, and shares a bearer secret between runner and cache-server via `ACT_RUNNER_EXTERNAL_SECRET`. Once that PR is merged and deployed, `cache: 'pip'` in this workflow will start hitting the cache and the wall-clock per job should drop from ~12 min to the ~3-4 min the PR description promises.
Author
Contributor

Opened admin/ansible#34 (supersedes #33) with the actual fix for the cache timeout — the first attempt accidentally pulled in hermes_squid/ changes from upstream. The new PR touches only poc/gitea/:

  • poc/gitea/stack-compose.yml: new cache_server service on the gitea overlay; shared secret mounted from Docker secret gitea_cache_secret (no more CHANGE_ME_SHARED_SECRET placeholder in env).
  • poc/gitea/config.yaml: cache.{host,port,external_server} point at gitea_cache_server:37051; external_secret empty because the secret is loaded from the file by ACT_RUNNER_EXTERNAL_SECRET_FILE.

Diff: +32/-19 across 2 files. Once #34 is merged and deployed, cache: 'pip' here should start hitting the cache.

→ admin/ansible#34

Opened **admin/ansible#34** (supersedes #33) with the actual fix for the cache timeout — the first attempt accidentally pulled in `hermes_squid/` changes from upstream. The new PR touches only `poc/gitea/`: - `poc/gitea/stack-compose.yml`: new `cache_server` service on the `gitea` overlay; shared secret mounted from Docker secret `gitea_cache_secret` (no more `CHANGE_ME_SHARED_SECRET` placeholder in env). - `poc/gitea/config.yaml`: `cache.{host,port,external_server}` point at `gitea_cache_server:37051`; `external_secret` empty because the secret is loaded from the file by `ACT_RUNNER_EXTERNAL_SECRET_FILE`. Diff: +32/-19 across 2 files. Once #34 is merged and deployed, `cache: 'pip'` here should start hitting the cache. → https://gitea.rudolf.keenetic.link/admin/ansible/pulls/34
hermes added 1 commit 2026-09-03 22:42:45 +02:00
ci: empty commit to retrigger cache verification run
Python package / build (3.12.3) (pull_request) Canceled after 0s
Python package / build (3.13) (pull_request) Canceled after 0s
Python package / build (3.14) (pull_request) Canceled after 0s
c6af53ec7e
hermes added 1 commit 2026-09-03 23:17:42 +02:00
ci: retrigger after runner cache register fix
Python package / build (3.12.3) (pull_request) Successful in 3m45s
Python package / build (3.13) (pull_request) Successful in 2m11s
Python package / build (3.14) (pull_request) Successful in 2m18s
0bbb3c4c26
hermes added 1 commit 2026-09-03 23:43:45 +02:00
ci: retrigger after run #58 success
Python package / build (3.12.3) (pull_request) Successful in 2m10s
Python package / build (3.13) (pull_request) Successful in 2m12s
Python package / build (3.14) (pull_request) Successful in 2m22s
ac682c1a92
Author
Contributor

Verification ✅ — admin/ansible#34 deploy completed; cache is reachable and the CI pipeline is ~5x faster.

Run Job wall-clock
#16 (before fix, cache ETIMEDOUT ~9 min/job) 3.12.3 12m 41s
3.13 11m 13s
3.14 11m 18s
total 35m 12s
#59 (after fix, cache works) 3.12.3 2m 10s
3.13 2m 12s
3.14 2m 22s
total 6m 44s

Speedup: ~5.2× per run (35m → 7m). Per job the wall-clock dropped from ~11-12 min to ~2-2.5 min, which matches PR #15's prediction (3-4 min) — we hit the optimistic end of the band because pip cache is fully warm.

Verified runs:

Infra fix: admin/ansible#34 — dedicated gitea_cache_server service on the gitea overlay, shared secret via Docker secret gitea_cache_secret (ACT_RUNNER_EXTERNAL_SECRET_FILE), entrypoint scripts that substitute the secret into a runtime copy of config.yaml's cache.external_secret.

**Verification ✅** — admin/ansible#34 deploy completed; cache is reachable and the CI pipeline is ~5x faster. | Run | Job | wall-clock | |-----|-----|------------| | **#16** (before fix, cache ETIMEDOUT ~9 min/job) | 3.12.3 | 12m 41s | | | 3.13 | 11m 13s | | | 3.14 | 11m 18s | | | **total** | **35m 12s** | | **#59** (after fix, cache works) | 3.12.3 | **2m 10s** | | | 3.13 | **2m 12s** | | | 3.14 | **2m 22s** | | | **total** | **6m 44s** | **Speedup: ~5.2× per run** (35m → 7m). Per job the wall-clock dropped from ~11-12 min to ~2-2.5 min, which matches PR #15's prediction (3-4 min) — we hit the optimistic end of the band because pip cache is fully warm. Verified runs: - https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16 (before) - https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/59 (after) Infra fix: https://gitea.rudolf.keenetic.link/admin/ansible/pulls/34 — dedicated `gitea_cache_server` service on the `gitea` overlay, shared secret via Docker secret `gitea_cache_secret` (`ACT_RUNNER_EXTERNAL_SECRET_FILE`), entrypoint scripts that substitute the secret into a runtime copy of config.yaml's `cache.external_secret`.
Author
Contributor

Tracking issue created for this fix and the upstream CI improvement it unblocks:

→ admin/ansible#35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup)

Closes nothing (this PR is what #35 tracks), but linked for cross-reference.

Tracking issue created for this fix and the upstream CI improvement it unblocks: → https://gitea.rudolf.keenetic.link/admin/ansible/issues/35 — fix(hermes_infra/gitea): dedicated act_runner cache-server (5x CI speedup) Closes nothing (this PR is what #35 tracks), but linked for cross-reference.
hermes added 1 commit 2026-09-04 08:15:27 +02:00
ci: trim explanatory comments from python-package.yml
Python package / build (3.12.3) (pull_request) Successful in 2m17s
Python package / build (3.13) (pull_request) Failing after 11m17s
Python package / build (3.14) (pull_request) Canceled after 11m58s
0cb62d233a
Drop the CI-history and per-option narration comments; keep only what is
needed for the workflow to be readable (job names, step names). Mirrors
the repo's compose/env/entrypoint-with-no-explanatory-comments style.
admin added 1 commit 2026-09-04 08:31:04 +02:00
Merge branch 'main' into feat/optimize-ci-build-time
Python package / build (3.12.3) (pull_request) Successful in 10m47s
Python package / build (3.13) (pull_request) Successful in 10m50s
Python package / build (3.14) (pull_request) Successful in 10m53s
e3bc8a3212
hermes added 1 commit 2026-09-04 09:31:55 +02:00
ci: retrigger after comment cleanup
Python package / build (3.12.3) (pull_request) Successful in 4m17s
Python package / build (3.13) (pull_request) Successful in 2m15s
Python package / build (3.14) (pull_request) Successful in 2m28s
43b5afaf94
admin merged commit 8059cc74ea into main 2026-09-04 09:48:31 +02:00
admin deleted branch feat/optimize-ci-build-time 2026-09-04 09:48:31 +02:00
Sign in to join this conversation.
No Reviewers
No labels
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/brencher#15