forked from admin/brencher
ci: speed up python-package.yml from ~13min to ~3-4min
The build job was running for ~13 minutes per PR even with tests disabled (per PR #13). The actual cause: the matrix had 3 python-versions x 5 checkers = 15 separate CI jobs, each re-installing the same requirements.txt + requirements-dev.txt from scratch with no pip cache. This commit: 1. Enables pip caching via actions/setup-python@v5 cache: 'pip' keyed on the contents of requirements.txt and requirements-dev.txt. When those files don't change, pip install is near-instant. When they do change, only the diff downloads. 2. Drops two debug-only matrix entries that were never real checks: python -c 'import sys; print(...)' (debug) pip list (debug) They took up two of the five matrix cells but produced no actionable signal. 3. Restructures the matrix so each Python version runs ONE job that does install + flake8 (2 invocations) + mypy sequentially, instead of one job per (version, checker) pair. Net: 3 jobs total instead of 15. 4. Adds a concurrency: cancel-in-progress block so a new push to the same PR auto-cancels the previous in-flight run, saving runner minutes when iterating on a fix. Verified: - YAML parses via PyYAML - jobs dict contains ['build'] only - matrix python versions: ['3.12.3', '3.13', '3.14'] - total CI jobs: 3 (was 15) - actions/setup-python@v5 with cache: 'pip' + cache-dependency-path on requirements.txt and requirements-dev.txt Expected wall-clock (rough estimate, depends on cache hits): Before: ~13 min (15 jobs, no pip cache, install on every job) After (cold): ~5-7 min (3 jobs, pip cache miss on first run) After (warm): ~3-4 min (3 jobs, pip cache hits) If caching doesn't activate (e.g. self-hosted runner ignores cache:), the win comes mostly from 15 -> 3 jobs (~5x faster). Note: PR #14 (re-enable tests) is still open. If/when it merges, this PR will need a follow-up to re-add the tests job with the same cache setup applied.
This commit is contained in:
@@ -1,5 +1,15 @@
|
||||
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
|
||||
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
|
||||
# CI workflow: lint + mypy on every push/PR to main.
|
||||
#
|
||||
# Optimization history:
|
||||
# - PR #13 commented out the tests: job (kept the build job).
|
||||
# - This commit makes the remaining build job actually fast:
|
||||
# 1. pip cache via actions/setup-python@v5 cache: 'pip'
|
||||
# 2. Dropped the debug-only matrix entries (python -c, pip list)
|
||||
# 3. Restructured the matrix so flake8 + mypy share one Python
|
||||
# install per Python version instead of spinning up a fresh
|
||||
# venv per (version, checker) tuple.
|
||||
#
|
||||
# Net effect: from ~13 minutes down to a target of ~3-4 minutes per run.
|
||||
|
||||
name: Python package
|
||||
|
||||
@@ -10,74 +20,49 @@ on:
|
||||
pull_request:
|
||||
branches: [ "main" ]
|
||||
|
||||
# Cancel in-progress runs on the same PR/branch when new commits land.
|
||||
concurrency:
|
||||
group: ci-${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python-version: [
|
||||
# "3.11.3",
|
||||
"3.12.3",
|
||||
"3.13",
|
||||
"3.14",
|
||||
# "3.15",
|
||||
]
|
||||
checker: [
|
||||
"python -c \"import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')\"",
|
||||
"pip list",
|
||||
"flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics",
|
||||
"flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics",
|
||||
# "ty check backend",
|
||||
"mypy --no-incremental --config-file mypy.ini backend tests",
|
||||
]
|
||||
python-version: ["3.12.3", "3.13", "3.14"]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up Python ${{ matrix.python-version }}, ${{ matrix.checker }}
|
||||
uses: actions/setup-python@v3
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
# Cache pip downloads keyed by the OS + the resolved requirements
|
||||
# file content. When requirements*.txt don't change, the cache
|
||||
# hits and pip install is near-instant. When they do change, only
|
||||
# the diff needs to download.
|
||||
cache: 'pip'
|
||||
cache-dependency-path: |
|
||||
requirements.txt
|
||||
requirements-dev.txt
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
pip install -r requirements-dev.txt
|
||||
- name: Run checker
|
||||
run: ${{ matrix.checker }}
|
||||
|
||||
# tests:
|
||||
# runs-on: ubuntu-latest
|
||||
# needs: build
|
||||
# strategy:
|
||||
# fail-fast: false
|
||||
# matrix:
|
||||
# python-version: [
|
||||
# # "3.11.3",
|
||||
# "3.12.3",
|
||||
# "3.13",
|
||||
# "3.14",
|
||||
# # "3.15",
|
||||
# ]
|
||||
# checker: [
|
||||
# "pytest",
|
||||
# ]
|
||||
# steps:
|
||||
# - uses: actions/checkout@v4
|
||||
# - name: Set up Python ${{ matrix.python-version }}, ${{ matrix.checker }}
|
||||
# uses: actions/setup-python@v3
|
||||
# with:
|
||||
# python-version: ${{ matrix.python-version }}
|
||||
# - name: Install dependencies
|
||||
# run: |
|
||||
# # sudo apt-get update
|
||||
# # sudo apt-get upgrade -y
|
||||
# # sudo apt-get install -y --no-install-recommends git docker.io
|
||||
# python -m pip install --upgrade pip
|
||||
# pip install -r requirements-dev.txt
|
||||
# playwright install --with-deps chromium
|
||||
# - name: Run checker
|
||||
# run: "pytest"
|
||||
#
|
||||
#
|
||||
#
|
||||
- name: Lint (flake8)
|
||||
# Single flake8 invocation covers both the syntax-only check
|
||||
# (E9,F63,F7,F82) and the complexity check (max-complexity 10).
|
||||
# flake8 with --select on the second pass exits non-zero only on
|
||||
# syntax errors; the complexity pass exits 0 always (--exit-zero).
|
||||
run: |
|
||||
flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics
|
||||
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
|
||||
|
||||
- name: Type-check (mypy)
|
||||
run: mypy --no-incremental --config-file mypy.ini backend tests
|
||||
|
||||
Reference in New Issue
Block a user