Author SHA1 Message Date
copilot-swe-agent[bot]andrudolf1 0e43c0fd13 harden playwright test config and assertions
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.12.3) (pull_request) Successful in 36s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.13) (pull_request) Successful in 34s
Python package / build (flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics, 3.14) (pull_request) Successful in 36s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.12.3) (pull_request) Successful in 33s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.13) (pull_request) Successful in 35s
Python package / build (flake8 . --count --select=E9,F63,F7,F82 --ignore=F824 --show-source --statistics, 3.14) (pull_request) Successful in 36s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.12.3) (pull_request) Successful in 1m43s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.13) (pull_request) Successful in 1m54s
Python package / build (mypy --no-incremental --config-file mypy.ini backend tests, 3.14) (pull_request) Successful in 1m59s
Python package / build (pip list, 3.12.3) (pull_request) Successful in 31s
Python package / build (pip list, 3.13) (pull_request) Successful in 33s
Python package / build (pip list, 3.14) (pull_request) Successful in 31s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.12.3) (pull_request) Successful in 32s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.13) (pull_request) Successful in 36s
Python package / build (python -c "import sys; print(f'Python {sys.version}'); print(f'Build: {sys.version_info}'); print(f'Implementation: {sys.implementation}')", 3.14) (pull_request) Successful in 31s
Python package / tests (pytest, 3.12.3) (pull_request) Canceled after 0s
Python package / tests (pytest, 3.13) (pull_request) Canceled after 0s
Python package / tests (pytest, 3.14) (pull_request) Canceled after 0s
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/d019d010-3ca0-4a73-8826-620eef58edab

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 20:08:12 +00:00
copilot-swe-agent[bot]andrudolf1 ab01fca841 stabilize dry-run playwright integration
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/d019d010-3ca0-4a73-8826-620eef58edab

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 20:06:11 +00:00
copilot-swe-agent[bot]andrudolf1 b2782ee220 clarify async taskgroup behavior
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:36:56 +00:00
copilot-swe-agent[bot]andrudolf1 1acea85934 add async shutdown cleanup and review fixes
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:35:59 +00:00
copilot-swe-agent[bot]andrudolf1 6f3dd6ebd8 refactor app concurrency to asyncio taskgroup
Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/9e719b58-ad2a-4eca-99bf-43c750f71a72

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-19 19:34:46 +00:00
copilot-swe-agent[bot] efe49119a3 Initial plan 2026-05-19 19:02:30 +00:00
admin 2a14d70fab playwright tests. refactor backend (#79)
* refactor backend
* Playwright test
2026-05-10 20:14:50 +02:00
admin 234ffdd634 rework secondaries (#78)
* rework secondaries
2026-05-09 17:31:50 +02:00
Copilotandrudolf1 c3e0bff795 Move slave connectivity to separate class with multi-connector support (#76)
* Initial plan

* Move slave connectivity to separate SlaveConnector class in backend/slave.py

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/8dab3353-0172-49d4-818c-cff27d5b13b5

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Support multiple slave connectors via Environment.slave_urls list

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/081649bc-80b1-407d-9895-cdd621727907

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Add slave_urls to brencher2/brencher_local2 configs; remove SLAVE_BRENCHER from docker-compose.yml

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/6f458415-0672-4842-9e0c-8670daf77283

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-05-09 11:44:18 +02:00
Copilot 40544a3e42 Rework desired branch resolution into a dedicated Git state step across all configs (#73)
all job
2026-04-29 23:17:59 +02:00
admin e44c78e569 Escape <> (#71) 2026-04-14 21:18:30 +02:00
adminandGitHub Actions 20e0eef4d1 Squash (#70)
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
2026-04-11 15:01:56 +02:00
admin 71453214b8 Reset caches. Spinner fix 2026-04-11 14:17:22 +02:00
902bd156d3 Add is_running spinner to pipeline step UI; improve Docker step implementations (#69)
* Initial plan

* Add step spinner: is_running field, backend emit, and frontend spinner UI

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/6e795096-6c09-458a-8778-a7997f5b2b9e

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* DockerComposeBuild returns image SHAs; DockerSwarmDeploy health check after deploy

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/1c22f6b5-4f50-44df-b4b8-43cafbcc647c

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Guard img.id for None in DockerComposeBuild; add type: ignore on websocket.state.last_payloads

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/645f064f-c7e1-4abf-b5ab-68fd500148ca

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-04-11 00:40:22 +02:00
Copilot e84c9fbb96 CachingStep: invalidate cache based on upstream input hash; per-client WebSocket deduplication (#66) 2026-04-10 23:52:25 +02:00
admin 7c20706a25 Brencher docker build cache (#67) 2026-04-10 22:20:15 +02:00
efef0b692f Add per-environment dry run toggle (#61)
* Initial plan

* Add dry run toggle button per environment with step highlighting

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/3e2e170a-f85b-43ed-b090-e348c3b3d444

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Remove dry_run_aware logic; keep only dry run toggle button per environment

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/19a9c77a-472b-4fa2-b23f-3e5b8258b209

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Replace dry run toggle with play/pause icon button, right-aligned

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/96e3dcfc-29e2-49cb-b199-6dba2520096c

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Always sync dry run button state from server environments update

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/93b06c01-a144-4de9-90de-dc7566afdd0c

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* x

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
Co-authored-by: rudolf <rudolfss13@gmail.com>
2026-04-06 23:25:27 +02:00
694283e7c4 Migrate to uvicorn: Flask/SocketIO → FastAPI + native WebSocket (#53)
* Initial plan

* Migrate from Flask/SocketIO to FastAPI/uvicorn with native WebSocket support

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Consolidate three WebSocket endpoints into single /ws connection

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Replace event/data envelope with named message fields

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* Fix bad merge: remove duplicate lines in app.py and restore forEach loop in app.js

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/ac246b64-cf2e-4cd3-ab28-5cd9caf28944

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

* x

* x

* x

* x

* x

* x

* x

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
Co-authored-by: rudolf <rudolfss13@gmail.com>
2026-03-31 16:48:03 +02:00
Copilotandrudolf1 313c34424a Custom columns: structured GitUnmerge result + dynamic branch table columns (#59)
* Initial plan

* Implement custom columns: GitUnmergeResult dataclass, dynamic UI column rendering, fix deployed branch display

Agent-Logs-Url: https://github.com/rudolf1/brencher/sessions/8a912a0d-e2fc-4107-a128-60e6623b6920

Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: rudolf1 <5525769+rudolf1@users.noreply.github.com>
2026-03-31 13:16:53 +02:00
admin d6977bad9d Gmail mcp (#57)
* gmail mcp

* x

* x

* x

* x

* x
2026-03-29 10:58:49 +02:00
45 changed files with 2379 additions and 877 deletions
@@ -0,0 +1,53 @@
---
name: playwright-brencher
description: "Write or refactor Playwright tests for brencher UI. Use when creating page helpers, extracting browser code from tests, or enforcing set_/verify_ method naming."
---
# Playwright Brencher Conventions
Use this skill when writing or refactoring Playwright tests in this repository.
## Required API Shape
Create and use a context manager in this style:
```python
with brencher_page(APP_URL) as page:
page.verify_dry_run_on()
page.set_dry_run_off()
page.verify_dry_run_off()
```
## Naming Rules
1. Action methods that change UI/application state must use `set_...` names.
2. Verification methods (assertions, waits, checks) must use `verify_...` names.
3. Keep low-level selector helpers private (prefix `_`).
4. Prefer one page-object class per screen/area (`BrencherPage` for the current app page).
## Structure Rules
1. Put shared Playwright helpers in `tests/playwright_helper.py`.
2. Keep tests focused on scenario flow; move DOM selectors and waiting logic into page helper methods.
3. Use context manager functions to own Playwright/browser lifecycle.
4. Inline trivial one-use logic where it improves readability and does not duplicate behavior.
## Method Guidance
Recommended style for page object methods:
- `set_*`: click/toggle/type actions.
- `verify_*`: assert visible state, button title/text, or wait until target state is reached.
Example method names:
- `set_dry_run_off`
- `verify_dry_run_on`
- `verify_dry_run_off`
## Review Checklist
1. Test uses `with brencher_page(...) as page` style.
2. No direct `sync_playwright()` block remains in the test body when helper abstraction is intended.
3. Action and verification methods follow `set_` / `verify_` naming.
4. Selector details are encapsulated in private helper methods.
@@ -48,6 +48,7 @@ jobs:
fi
echo "=== Processing branch: $br ==="
# Create/update a local branch from remote
git reset --hard HEAD
git checkout -B "$br" "origin/$br"
# Merge origin/main into branch (non-interactive)
if git merge --no-edit --squash "origin/$MAIN"; then
+37 -4
View File
@@ -12,13 +12,12 @@ on:
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: [
# "3.11",
# "3.11.3",
"3.12.3",
"3.13",
"3.14",
@@ -31,7 +30,6 @@ jobs:
"flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics",
# "ty check backend",
"mypy --no-incremental --config-file mypy.ini backend tests",
"pytest --ignore=tests/integration",
]
steps:
@@ -43,7 +41,42 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install flake8 pytest ty
pip install -r requirements.txt
pip install -r requirements-dev.txt
- name: Run checker
run: ${{ matrix.checker }}
tests:
runs-on: ubuntu-latest
needs: build
strategy:
fail-fast: false
matrix:
python-version: [
# "3.11.3",
"3.12.3",
"3.13",
"3.14",
# "3.15",
]
checker: [
"pytest",
]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}, ${{ matrix.checker }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
# sudo apt-get update
# sudo apt-get upgrade -y
# sudo apt-get install -y --no-install-recommends git docker.io
python -m pip install --upgrade pip
pip install -r requirements-dev.txt
playwright install --with-deps chromium
- name: Run checker
run: "pytest"
+30 -3
View File
@@ -1,16 +1,40 @@
# Use official Python image with uv pre-installed
FROM python:3.12.3-alpine
FROM python:3.12.3-slim AS typecheck
# Install uv (if not present)
RUN pip install uv
# Set workdir
WORKDIR /app
# Install development dependencies and run mypy before the runtime image is built.
COPY requirements.txt requirements-dev.txt mypy.ini ./
RUN uv venv .venv && \
uv pip install -r requirements-dev.txt
COPY backend/ ./backend/
COPY tests/ ./tests/
RUN . .venv/bin/activate && \
mypy --no-incremental --config-file mypy.ini backend tests && \
touch /tmp/typecheck.done
# Use official Python image with uv pre-installed
FROM python:3.12.3-alpine AS base
# Install uv (if not present)
RUN pip install uv
# Install git
RUN apk update && apk upgrade
RUN apk add --no-cache git docker-cli
# Set workdir
WORKDIR /app
# Copy requirements and install dependencies with uv
COPY requirements.txt ./
RUN uv venv .venv && \
uv pip install -r requirements.txt
@@ -18,12 +42,15 @@ RUN uv venv .venv && \
COPY backend/ ./backend/
COPY frontend/ ./frontend/
# Expose port for Flask
# Expose port for the web server
EXPOSE 5001
# Ensure the mypy pre-build stage is executed.
COPY --from=typecheck /tmp/typecheck.done /tmp/typecheck.done
# Set environment variables
ENV PYTHONUNBUFFERED=1
ENV PATH="/app/.venv/bin:$PATH"
# Start the Flask app
# Start the app with uvicorn
CMD ["python3", "backend/app.py"]
+25
View File
@@ -0,0 +1,25 @@
FROM nginx:1.27-alpine
# Configure nginx to listen on 5001 to match container port expectations.
RUN sed -i 's/listen 80;/listen 5001;/' /etc/nginx/conf.d/default.conf
# Build a custom landing page directly in the image.
RUN set -eux; \
{ \
echo '<!doctype html>'; \
echo '<html lang="en">'; \
echo '<head>'; \
echo ' <meta charset="utf-8">'; \
echo ' <meta name="viewport" content="width=device-width, initial-scale=1">'; \
echo ' <title>Brencher Test App</title>'; \
echo '</head>'; \
echo '<body>'; \
echo ' <h1>Nginx Is Running</h1>'; \
echo ' <p>Custom landing page for the Brencher test app container.</p>'; \
echo ' <p>Port: 5001</p>'; \
echo ' <p>Git revision: unknown</p>'; \
echo '</body>'; \
echo '</html>'; \
} > /usr/share/nginx/html/index.html
EXPOSE 5001
+153 -372
View File
@@ -1,28 +1,21 @@
import json
import asyncio
import logging
import os
import threading
import time
import signal
import sys
import traceback
import types
from dataclasses import asdict, replace
from typing import List, Dict, Any, Optional
from typing import TypeVar
from typing import Any, Callable, Dict, List, Optional
import socketio as socketio_client
from discover_envs import build_environments, parse_arguments
from utils import sigchld_handler
from dotenv import load_dotenv
from flask import Flask, send_from_directory
from flask.json.provider import DefaultJSONProvider
from flask_socketio import SocketIO, emit
from enironment import Environment, wrap_in_cached
from processing import reset_caches
from enironment import Environment, wrap_in_cached, SharedStateHolder, get_step
from steps.git import GitClone
from steps.step import CachingStep
# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logging.getLogger('werkzeug').setLevel(logging.WARNING)
logger = logging.getLogger(__name__)
# Load environment variables from .env file
@@ -30,375 +23,153 @@ load_dotenv("local.env")
load_dotenv('/run/secrets/brencher-secrets')
class DataclassJSONEncoder(json.JSONEncoder):
def default(self, o: Any) -> Any:
if hasattr(o, '__dataclass_fields__'):
return asdict(o)
if isinstance(o, BaseException):
return str(o)
try:
return super().default(o)
except TypeError:
return str(o)
custom_json = types.SimpleNamespace()
custom_json.dumps = lambda obj, **kwargs: json.dumps(obj, cls=DataclassJSONEncoder, **kwargs)
custom_json.loads = json.loads
class DataclassJSONProvider(DefaultJSONProvider):
def dumps(self, obj: Any, **kwargs: Any) -> str:
return json.dumps(obj, cls=DataclassJSONEncoder, **kwargs)
def loads(self, s: str | bytes, **kwargs: Any) -> Any:
return json.loads(s, **kwargs)
app = Flask(__name__, static_folder='frontend')
app.json = DataclassJSONProvider(app)
socketio = SocketIO(app, cors_allowed_origins="*", json=custom_json)
FRONTEND_DIR = os.path.join(os.path.dirname(__file__), '../frontend')
# In-memory state
environments: Dict[str, Environment] = {}
environments_slaves: Dict[str, Environment] = {}
branches_slaves: Dict[str, Dict[str, Any]] = {}
state_lock = threading.Lock()
@app.route('/')
def serve_index() -> Any:
return send_from_directory(FRONTEND_DIR, 'index.html')
@app.route('/<path:path>')
def serve_static(path: str) -> Any:
return send_from_directory(FRONTEND_DIR, path)
# --- WebSocket Endpoints ---
from flask_socketio import Namespace
T = TypeVar('T')
def merge_dicts(a: Dict[str, T], b: Dict[str, T]) -> Dict[str, T]:
result: Dict[str, T] = {}
for k in (a.keys() | b.keys()):
if k not in a or k not in b:
result[k] = a.get(k, b.get(k)) # type: ignore[assignment]
continue
if (
isinstance(a[k], dict)
and isinstance(b[k], dict)
):
result[k] = merge_dicts(a[k], b[k]) # type: ignore[arg-type,assignment]
else:
result[k] = b[k]
return result
slave_url = os.getenv('SLAVE_BRENCHER')
remote_sio = None
if slave_url:
logger.info(f"SLAVE_BRENCHER set, will connect to master at {slave_url}")
remote_sio = socketio_client.Client(logger=False, engineio_logger=False)
# Handlers for events from master -> re-emit locally (marked so we don't loop)
@remote_sio.on('branches', namespace='/ws/branches')
def _remote_branches(data: Any) -> None:
global branches_slaves
branches_slaves = data
try:
socketio.emit('branches', get_global_branches_to_emit(), namespace='/ws/branches')
except Exception as e:
logger.error(f"Error forwarding remote branches locally: {e}")
@remote_sio.on('environments', namespace='/ws/environment')
def _remote_environments(data: Any) -> None:
global environments_slaves
environments_slaves = data
try:
socketio.emit('environments', get_global_envs_to_emit(), namespace='/ws/environment')
except Exception as e:
logger.error(f"Error forwarding remote environments locally: {e}")
@remote_sio.on('error', namespace='/ws/errors')
def _remote_error(data: Any) -> None:
try:
socketio.emit('error', data, namespace='/ws/errors')
except Exception as e:
logger.error(f"Error forwarding remote errors locally: {e}")
@remote_sio.event
def connect() -> None:
logger.info("Connected to master brencher (SLAVE mode).")
@remote_sio.event
def disconnect() -> None:
logger.info("Disconnected from master brencher (SLAVE mode).")
class BranchesNamespace(Namespace):
def on_connect(self, auth: Optional[Any] = None) -> None:
emit('branches', get_global_branches_to_emit())
def on_update(self, data: Any) -> None:
emit('branches', get_global_branches_to_emit())
def get_local_envs_to_emit() -> Dict[str, Dict[str, Any]]:
env_dtos: Dict[str, Dict[str, Any]] = {}
for env in environments.values():
pipeline_state: List[Dict[str, Any]] = []
for r in env.pipeline:
try:
if isinstance(r, CachingStep):
result = r._result
else:
result = r.progress()
if isinstance(result, BaseException):
stack = traceback.format_exception(type(result), result, result.__traceback__)
pipeline_state.append({
"name": r.name,
"status": [str(result), stack],
"error": True,
})
else:
pipeline_state.append({
"name": r.name,
"status": result,
})
except BaseException as e:
stack = traceback.format_exception(type(e), e, e.__traceback__)
pipeline_state.append({
"name": r.name,
"status": [str(e), stack],
"error": True,
})
env_dtos[env.id] = asdict(replace(env, pipeline=[]))
env_dtos[env.id]['pipeline'] = pipeline_state
return env_dtos
def get_global_envs_to_emit() -> Any:
global environments, environments_slaves
local_envs = get_local_envs_to_emit()
merge_result = merge_dicts(local_envs, environments_slaves) # type: ignore[misc]
# logger.info(f"Local keys: {local_envs.keys()}")
# logger.info(f"Slave keys: {environments_slaves.keys()}")
common_keys = set(local_envs.keys()) & set(environments_slaves.keys())
if len(common_keys) > 0:
socketio.emit('error', {'message': f"Conflict: both master and slave have environment with id {common_keys}"},
namespace='/ws/errors')
return merge_result
def get_local_branches_to_emit() -> Dict[str, Dict[str, List[Any]]]:
global environments
branches: Dict[str, Dict[str, List[Any]]] = {}
for k, env in environments.items():
branches[k] = {}
try:
for step in env.pipeline:
if isinstance(step, GitClone):
branches[k] = {**step.get_branches()}
if isinstance(step, CachingStep) and isinstance(step.step, GitClone):
branches[k] = {**step.step.get_branches()}
except BaseException as e:
logger.error(f"Error fetching branches for environment {env.id}: {str(e)}")
return branches
def get_global_branches_to_emit() -> Dict[str, Dict[str, List[Any]]]:
global branches_slaves
local_branches: Dict[str, Dict[str, List[Any]]] = get_local_branches_to_emit()
return merge_dicts(local_branches, branches_slaves)
environment_update_event = threading.Event()
@app.route('/state')
def serve_state() -> Any:
return get_global_envs_to_emit()
@app.route('/branches')
def serve_branches() -> Any:
return get_global_branches_to_emit()
class EnvironmentNamespace(Namespace):
def on_connect(self, auth: Optional[Any] = None) -> None:
emit('environments', get_global_envs_to_emit())
def on_update(self, data: Dict[str, Any]) -> None:
global environments, remote_sio
logger.info(f"Received environment update: {data}")
if data.get('id') == '':
reset_caches(list(environments.values()))
else:
for env in environments.values():
if env.id == data.get('id'):
env.branches = data.get('branches', env.branches)
logger.info(f"Updated environment {env.id} branches to {env.branches}")
environment_update_event.set()
if remote_sio and remote_sio is not None and remote_sio.connected:
remote_sio.emit('update', data, namespace='/ws/environment')
logger.info(f"Updated slave")
emit('environments', get_global_envs_to_emit(), namespace='/ws/environment')
class ErrorsNamespace(Namespace):
def on_connect(self, auth: Optional[Any] = None) -> None:
pass
def on_error(self, data: Any) -> None:
emit('error', data)
socketio.on_namespace(BranchesNamespace('/ws/branches'))
socketio.on_namespace(EnvironmentNamespace('/ws/environment'))
socketio.on_namespace(ErrorsNamespace('/ws/errors'))
def sigchld_handler(signum, frame): # type: ignore[no-untyped-def]
"""Reap zombie processes"""
while True:
try:
pid, status = os.waitpid(-1, os.WNOHANG)
if pid == 0:
break
except ChildProcessError:
break
import signal
class App:
def __init__(self, cli_env_ids_str: str, dry_run: bool = False) -> None:
global environments
# Register the handler
signal.signal(signal.SIGCHLD, sigchld_handler)
def __init__(self, environments: Dict[str, Environment]) -> None:
self.environments: Dict[str, Environment] = {id: wrap_in_cached(e) for id, e in environments.items()}
self.environment_update_event = asyncio.Event()
self.environment_update_version = 0
self.emit_callback: Optional[Callable[[], None]] = None
self.reset_requested = False
self.shutdown_event = asyncio.Event()
self.web_app: Any = None
import configs.brencher
import configs.brencher2
import configs.brencher_local2
import configs.brencher_local1
import configs.torrserv_proxy
import configs.immich
import configs.registry
environments_l: List[Environment] = [
configs.brencher.brencher,
configs.brencher2.brencher2,
configs.brencher_local2.brencher_local2,
configs.brencher_local1.brencher_local1,
configs.torrserv_proxy.torrserv_proxy,
configs.immich.immich,
configs.registry.registry,
]
environments = {e.id: e for e in environments_l}
logger.info(f"cli_env_ids {cli_env_ids_str}")
if len(cli_env_ids_str) > 0 and cli_env_ids_str[0] == '-':
cli_env_ids = cli_env_ids_str[1:].split(',')
cli_env_ids = [x for x in cli_env_ids if len(x) > 0]
logger.info(f"cli_env_ids (minus) {cli_env_ids}")
if cli_env_ids and len(cli_env_ids) > 0:
environments = {k: e for k, e in environments.items() if k not in cli_env_ids}
else:
cli_env_pairs = {x[0]: x[1] if len(x) > 1 else None for x in [x.split(":")
for x in cli_env_ids_str.split(',')
if len(x) > 0
]
}
logger.info(f"cli_env_ids {cli_env_pairs}")
if cli_env_pairs and len(cli_env_pairs) > 0:
environments = {k: e for k, e in environments.items() if k in cli_env_pairs.keys()}
for k, v in cli_env_pairs.items():
if v is not None:
if k in environments:
env = environments[k]
env.branches = [(v, 'HEAD')]
logger.info(f"Overriding environment {k} branches to {env.branches}")
else:
logger.warning(f"Environment {k} not found to override branches")
def get_local_envs_to_emit(self) -> Dict[str, Dict[str, Any]]:
env_dtos: Dict[str, Dict[str, Any]] = {}
for env in self.environments.values():
pipeline_state: List[Dict[str, Any]] = []
for r in env.pipeline:
try:
if isinstance(r, CachingStep):
result = r._result
else:
logger.info(f"No branch override for environment {k}")
if dry_run:
for id, e in environments.items():
e.dry = True
result = r.progress()
logger.info(f"Resulting profiles {environments.keys()}")
environments = {id: wrap_in_cached(e) for id, e in environments.items()}
def processing_thread(self) -> None:
while True:
import processing
def emit_envs() -> None:
try:
socketio.emit('branches', get_global_branches_to_emit(), namespace='/ws/branches')
except Exception as e:
logger.error(f"Error emitting branches: {str(e)}")
try:
socketio.emit('environments', get_global_envs_to_emit(), namespace='/ws/environment')
except Exception as e:
logger.error(f"Error emitting environments: {str(e)}")
with state_lock:
logger.info(f"Processing")
if processing.process_all_jobs(list(environments.values()), lambda: emit_envs()):
environment_update_event.wait(timeout=1 * 5)
else:
environment_update_event.wait(timeout=1 * 60)
environment_update_event.clear()
def _connect_remote(self) -> None:
global remote_sio
print("Connecting to SLAVE_BRENCHER...")
while True:
if isinstance(result, BaseException):
stack = traceback.format_exception(type(result), result, result.__traceback__)
pipeline_state.append({
"name": r.name,
"status": [str(result), stack],
"error": True,
"is_running": True,
})
else:
pipeline_state.append({
"name": r.name,
"status": result,
"is_running": False,
})
except BaseException as e:
stack = traceback.format_exception(type(e), e, e.__traceback__)
pipeline_state.append({
"name": r.name,
"status": [str(e), stack],
"error": True,
"is_running": True,
})
env_dtos[env.id] = {'id': env.id }
env_dtos[env.id]['pipeline'] = pipeline_state
try:
if remote_sio is not None and not remote_sio.connected:
remote_sio.connect(slave_url, namespaces=['/ws/branches', '/ws/environment', '/ws/errors'])
except Exception as e:
logger.error(f"Could not connect to SLAVE_BRENCHER {slave_url}: {e}")
time.sleep(60)
shared_state = env.state.progress()
env_dtos[env.id]['branches'] = shared_state.branches
env_dtos[env.id]['branches_token'] = shared_state.token
env_dtos[env.id]['dry'] = shared_state.dry
except BaseException:
pass
return env_dtos
def run(self) -> None:
processing = threading.Thread(target=self.processing_thread)
processing.daemon = True
processing.start()
def get_local_branches_to_emit(self) -> Dict[str, Dict[str, List[Any]]]:
branches: Dict[str, Dict[str, List[Any]]] = {}
for k, env in self.environments.items():
branches[k] = {}
try:
step = get_step(env.pipeline, GitClone)
branches[k] = {**step.get_branches()}
except BaseException as e:
stack = traceback.format_exception(type(e), e, e.__traceback__)
logger.error(f"Error fetching branches for environment {env.id}: {str(e)}\n{''.join(stack)}")
t = threading.Thread(target=self._connect_remote, daemon=True)
t.start()
return branches
async def processing_loop(self) -> None:
while not self.shutdown_event.is_set():
import processing
emit = self.emit_callback or (lambda: None)
update_version = self.environment_update_version
logger.info("Processing")
if self.reset_requested:
await asyncio.to_thread(processing.reset_caches, list(self.environments.values()))
self.reset_requested = False
has_error = await asyncio.to_thread(processing.process_all_jobs, list(self.environments.values()), emit)
if self.shutdown_event.is_set():
return
# Skip the wait when a newer update arrived while the current processing pass was running.
if self.environment_update_version != update_version:
continue
timeout = 5 if has_error else 60
try:
await asyncio.wait_for(self.environment_update_event.wait(), timeout=timeout)
except asyncio.TimeoutError:
pass
self.environment_update_event.clear()
def notify_environment_update(self) -> None:
self.environment_update_version += 1
self.environment_update_event.set()
def request_reset(self) -> None:
self.reset_requested = True
def stop(self) -> None:
self.shutdown_event.set()
self.notify_environment_update()
if self.web_app is not None:
self.web_app.stop()
async def runAsync(self) -> None:
await self.processing_loop()
async def runHeadlessAsync(self) -> None:
await self.processing_loop()
def runHeadless(self) -> None:
"""Run the processing loop until cancelled; blocks forever."""
asyncio.run(self.runHeadlessAsync())
async def _run_web_server(self, web_app: Any) -> None:
"""Raise a sentinel exception when the server stops so TaskGroup cancels processing."""
await web_app.start_async()
raise _WebServerStopped()
async def runWebAsync(self, port: int) -> None:
import web
web_app = web.WebApp(core=self, port=port)
self.web_app = web_app
self.emit_callback = web_app.emit_envs
try:
# Keep processing and web serving coupled so shutdown or fatal failure tears down the whole app.
async with asyncio.TaskGroup() as task_group:
task_group.create_task(self.processing_loop())
task_group.create_task(self._run_web_server(web_app))
except* _WebServerStopped:
pass
finally:
self.emit_callback = None
self.web_app = None
def runWeb(self, port: int) -> None:
self.run()
socketio.run(app, host='0.0.0.0', port=port, debug=False, allow_unsafe_werkzeug=True)
asyncio.run(self.runWebAsync(port))
def run(self) -> None:
asyncio.run(self.runAsync())
class _WebServerStopped(Exception):
pass
if __name__ == '__main__':
import os
import sys
cli_env_ids_list = sys.argv[1:]
cli_env_ids_str: str
if len(cli_env_ids_list) == 0:
@@ -406,5 +177,15 @@ if __name__ == '__main__':
else:
cli_env_ids_str = cli_env_ids_list[0]
app1 = App(cli_env_ids_str, 'dry' in sys.argv[1:])
app1.runWeb(port=(5007 if 'noweb' in sys.argv[1:] else 5001))
signal.signal(signal.SIGCHLD, sigchld_handler)
environments = build_environments(cli_env_ids_str)
if 'dry' in sys.argv[1:]:
for e in environments.values():
e.state.set_dry(True)
app1 = App(environments)
if 'noweb' in sys.argv[1:]:
app1.runHeadless()
else:
app1.runWeb(5001)
+14 -8
View File
@@ -3,9 +3,19 @@ from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
clone = GitClone()
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
@@ -17,15 +27,13 @@ buildDocker = DockerComposeBuild(clone,
docker_compose_path="docker-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
build_cache = True,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
"user_group": "1000:137"
},
)
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher",
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=buildDocker,
@@ -36,7 +44,6 @@ deployDocker = DockerSwarmDeploy(
stack_name="brencher",
docker_compose_path="docker-compose.yml",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
checkPing = UrlCheck(
url="https://brencher.rudolf.keenetic.link/state",
@@ -52,11 +59,10 @@ logUrls = SimpleLog(message={
__all__ = ["brencher"]
brencher = Environment(
id="brencher",
branches=[],
dry=False,
repo="https://github.com/rudolf1/brencher.git",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
buildDocker,
dockerSwarmCheck,
+16 -9
View File
@@ -5,6 +5,7 @@ from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
def checkPingF(obj: Any) -> None:
@@ -20,8 +21,18 @@ def checkPingF(obj: Any) -> None:
raise Exception(f"brencher2 check failed for: {v}")
clone = GitClone()
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher2",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
@@ -33,15 +44,13 @@ buildDocker = DockerComposeBuild(clone,
docker_compose_path="docker-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
build_cache = True,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
"user_group": "1000:137"
},
)
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher2",
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=buildDocker,
@@ -62,7 +71,6 @@ deployDocker = DockerSwarmDeploy(
stack_name="brencher2",
docker_compose_path="docker-compose.yml",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
checkPing = UrlCheck(
url="https://brencher.rudolf.keenetic.link/state",
@@ -81,11 +89,10 @@ __all__ = ["brencher2"]
brencher2 = Environment(
id="brencher2",
branches=[],
dry=False,
repo="https://github.com/rudolf1/brencher.git",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
buildDocker,
dockerSwarmCheck,
@@ -93,5 +100,5 @@ brencher2 = Environment(
deployDocker,
checkPing,
logUrls
]
],
)
+14 -11
View File
@@ -1,9 +1,20 @@
from enironment import Environment
from steps.docker_plain import DockerImageBuild, DockerContainerCheck, DockerContainerDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
container_check = DockerContainerCheck(
container_name="brencher_plain-container"
)
unmerge = GitUnmerge(wd=clone, check=container_check)
state = SharedStateHolderInMemory(unmerge=unmerge)
clone = GitClone()
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
@@ -18,11 +29,6 @@ image_build = DockerImageBuild(
nocache=False,
)
# Step 2: Check if container already exists
container_check = DockerContainerCheck(
container_name="brencher_plain-container"
)
# Step 3: Deploy the container
container_deploy = DockerContainerDeploy(
image_build=image_build,
@@ -32,16 +38,13 @@ container_deploy = DockerContainerDeploy(
restart_policy={"Name": "unless-stopped"},
)
unmerge = GitUnmerge(wd=clone, check=container_check)
__all__ = ["brencher_local1"]
brencher_local1 = Environment(
id="brencher_local1",
branches=[("main", "HEAD")],
dry=False,
repo="https://github.com/rudolf1/brencher.git",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
image_build,
container_deploy,
+12 -8
View File
@@ -2,9 +2,18 @@ from enironment import Environment
from steps.docker import DockerComposeBuild
from steps.docker import DockerSwarmDeploy, DockerSwarmCheck
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher_local2",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
clone = GitClone()
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
@@ -22,9 +31,6 @@ buildDocker = DockerComposeBuild(clone,
},
)
dockerSwarmCheck = DockerSwarmCheck(
stack_name="brencher_local2",
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=buildDocker,
@@ -46,16 +52,14 @@ deployDocker = DockerSwarmDeploy(
stack_name="brencher_local2",
docker_compose_path="docker-compose.yml",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
__all__ = ["brencher_local2"]
brencher_local2 = Environment(
id="brencher_local2",
branches=[],
dry=False,
repo="https://github.com/rudolf1/brencher.git",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
buildDocker,
dockerSwarmCheck,
+77
View File
@@ -0,0 +1,77 @@
from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="immoscout")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="gmail_mcp",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
buildDocker = DockerComposeBuild(clone,
docker_repo_username="",
docker_repo_password="",
docker_compose_path="gmail-mcp-server/docker-compose.yml",
docker_repo_url="https://registry.rudolf.keenetic.link",
publish=False,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=buildDocker,
stackChecker=dockerSwarmCheck,
envs=lambda: {
"version": "auto-" + checkoutMerged.progress().version,
},
stack_name="gmail_mcp",
docker_compose_path="gmail-mcp-server/docker-compose.yml",
)
checkPing1 = UrlCheck(
url="http://100.70.193.97:3000/health",
expected={ "ok": True, "service": "gmail-mcp-server" },
)
checkPing2 = UrlCheck(
url="http://100.70.193.97:3001/health",
expected={ "ok": True, "service": "telegram-mcp-server" },
)
logUrls = SimpleLog(message={
"userLinks": {
"AppGm": "http://100.70.193.97:3000/health",
"AppTg": "http://100.70.193.97:3001/health",
}
})
__all__ = ["gmail_mcp"]
gmail_mcp = Environment(
id="gmail_mcp",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
buildDocker,
dockerSwarmCheck,
unmerge,
deployDocker,
checkPing1,
checkPing2,
logUrls
]
)
+14 -10
View File
@@ -2,17 +2,23 @@ from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(branchNamePrefix="ansible")
checkoutMerged = CheckoutMerged(clone,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="immich",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
deployDocker = DockerSwarmDeploy(
wd=clone,
@@ -24,7 +30,6 @@ deployDocker = DockerSwarmDeploy(
stack_name="immich",
docker_compose_path="poc/immich/stack-compose.yml",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
checkPing = UrlCheck(
url="https://immich.rudolf.keenetic.link/api/server/ping",
@@ -39,11 +44,10 @@ logUrls = SimpleLog(message={
__all__ = ["immich"]
immich = Environment(
id="immich",
branches=[("ansible/master", "HEAD")],
dry=False,
repo="https://github.com/rudolf1/uber_backup.git",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
+14 -10
View File
@@ -2,17 +2,23 @@ from enironment import Environment
from steps.checks import SimpleLog, UrlCheck
from steps.docker import DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(branchNamePrefix="ansible")
checkoutMerged = CheckoutMerged(clone,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
clone = GitClone(url="https://github.com/rudolf1/uber_backup.git", branchNamePrefix="ansible")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="registry",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
)
deployDocker = DockerSwarmDeploy(
wd=clone,
@@ -24,7 +30,6 @@ deployDocker = DockerSwarmDeploy(
stack_name="registry",
docker_compose_path="docker-compose-registry.yaml",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
checkPing = UrlCheck(
url="https://registry.rudolf.keenetic.link/v2/",
@@ -39,11 +44,10 @@ logUrls = SimpleLog(message={
__all__ = ["registry"]
registry = Environment(
id="registry",
branches=[("ansible/master", "HEAD")],
dry=False,
repo="https://github.com/rudolf1/uber_backup.git",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
dockerSwarmCheck,
unmerge,
+13 -8
View File
@@ -1,9 +1,19 @@
from enironment import Environment
from steps.docker import DockerComposeBuild, DockerSwarmCheck, DockerSwarmDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
clone = GitClone(url="https://github.com/rudolf1/reverseproxy.git")
dockerSwarmCheck = DockerSwarmCheck(
stack_name="torrserv_proxy",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
state = SharedStateHolderInMemory(unmerge=unmerge)
clone = GitClone()
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="brencher_bot"
@@ -20,9 +30,6 @@ buildDocker = DockerComposeBuild(clone,
},
)
dockerSwarmCheck = DockerSwarmCheck(
stack_name="torrserv_proxy",
)
deployDocker = DockerSwarmDeploy(
wd=clone,
buildDocker=buildDocker,
@@ -33,16 +40,14 @@ deployDocker = DockerSwarmDeploy(
stack_name="torrserv_proxy",
docker_compose_path="docker-compose.yml",
)
unmerge = GitUnmerge(clone, dockerSwarmCheck)
__all__ = ["torrserv_proxy"]
torrserv_proxy = Environment(
id="torrserv_proxy",
branches=[],
dry=False,
repo="https://github.com/rudolf1/reverseproxy.git",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
buildDocker,
dockerSwarmCheck,
+71
View File
@@ -0,0 +1,71 @@
import importlib
import logging
import pkgutil
from dataclasses import dataclass, field
from typing import Dict, List, Optional
from enironment import Environment, SharedStateHolder
from steps.step import CachingStep
logger = logging.getLogger(__name__)
@dataclass
class ParsedEnvArgs:
excludes: List[str] = field(default_factory=list)
includes: Dict[str, Optional[str]] = field(default_factory=dict)
def parse_arguments(cli_env_ids_str: str) -> ParsedEnvArgs:
logger.info(f"cli_env_ids {cli_env_ids_str}")
if len(cli_env_ids_str) > 0 and cli_env_ids_str[0] == '-':
excludes = [x for x in cli_env_ids_str[1:].split(',') if len(x) > 0]
logger.info(f"cli_env_ids (minus) {excludes}")
return ParsedEnvArgs(excludes=excludes)
includes: Dict[str, Optional[str]] = {
x[0]: x[1] if len(x) > 1 else None
for x in [x.split(":") for x in cli_env_ids_str.split(',') if len(x) > 0]
}
logger.info(f"cli_env_ids {includes}")
return ParsedEnvArgs(includes=includes)
def _discover_environments() -> Dict[str, Environment]:
import configs
found: Dict[str, Environment] = {}
seen: set[int] = set()
for module_info in pkgutil.iter_modules(configs.__path__, prefix=f"{configs.__name__}."):
module = importlib.import_module(module_info.name)
for value in vars(module).values():
if isinstance(value, Environment) and id(value) not in seen:
seen.add(id(value))
found[value.id] = value
return found
def build_environments(cli_env_ids_str: str) -> Dict[str, Environment]:
environments: Dict[str, Environment] = _discover_environments()
args: ParsedEnvArgs = parse_arguments(cli_env_ids_str)
if args.excludes:
environments = {k: e for k, e in environments.items() if k not in args.excludes}
elif args.includes:
environments = {k: e for k, e in environments.items() if k in args.includes}
for k, v in args.includes.items():
if v is None:
logger.info(f"No branch override for environment {k}")
continue
if k not in environments:
logger.warning(f"Environment {k} not found to override branches")
continue
env = environments[k]
for step in env.pipeline:
resolve_step = step
if isinstance(step, CachingStep):
resolve_step = step._step
if isinstance(resolve_step, SharedStateHolder):
resolve_step.set_branches([(v, 'HEAD')])
logger.info(f"Overriding environment {k} branches to {(v, 'HEAD')}")
logger.info(f"Resulting profiles {environments.keys()}")
return environments
+45 -9
View File
@@ -3,7 +3,7 @@ from __future__ import annotations
import logging
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List, Tuple
from typing import List, runtime_checkable, Protocol, Tuple, TypeAlias, Any
from typing import TypeVar
logger = logging.getLogger(__name__)
@@ -14,15 +14,43 @@ T = TypeVar('T')
@dataclass
class Environment:
id: str
branches: List[Tuple[str, str]] # List of [branch_name, desired_commit] pairs
dry: bool
repo: str # git repo
state: SharedStateHolder
pipeline: List[AbstractStep]
def __post_init__(self) -> None:
for p in self.pipeline:
p.env = self
@property
def dry(self) -> bool:
"""Get the dry run state from the shared state."""
return self.state.progress().dry
@dataclass
class SharedState:
branches: List[Tuple[str, str]]
dry: bool
token: str | None = None
class SharedStateConflictError(BaseException):
def __init__(self, message: str) -> None:
super().__init__(message)
@runtime_checkable
class SharedStateHolder(Protocol):
def set_branches(self, branches: List[Tuple[str, str]], expected_token: str | None = None) -> SharedState:
pass
def set_dry(self, dry: bool, expected_token: str | None = None) -> SharedState:
pass
def progress(self) -> SharedState:
pass
class AbstractStep[T](ABC):
_env: Environment | None = None
@@ -55,16 +83,24 @@ def wrap_in_cached(e: Environment) -> Environment:
from dataclasses import replace
from steps.step import CachingStep
result = replace(e, pipeline=[CachingStep(step) for step in e.pipeline])
cached = {it.step: it for it in result.pipeline if isinstance(it, CachingStep)}
cached = {it._step: it for it in result.pipeline if isinstance(it, CachingStep)}
for step in result.pipeline:
if isinstance(step, CachingStep):
step.step.env = result
for attr_name in dir(step.step):
step._step.env = result
for attr_name in dir(step._step):
if not attr_name.startswith('_'):
attr_value = getattr(step.step, attr_name)
attr_value = getattr(step._step, attr_name)
if isinstance(attr_value, AbstractStep):
cached_step = cached.get(attr_value)
if cached_step:
setattr(step.step, attr_name, cached_step)
setattr(step._step, attr_name, cached_step)
return result
def get_step(env: List[AbstractStep[Any]], class_or_tuple: type[T]) -> T:
from steps.step import CachingStep
for step in env:
if isinstance(step, class_or_tuple) or (isinstance(step, CachingStep) and isinstance(step._step, class_or_tuple)):
return step # type: ignore[return-value]
raise BaseException(f"Not step found {class_or_tuple}")
+1 -10
View File
@@ -3,7 +3,6 @@ import time
from typing import List, Callable
from enironment import Environment
from steps.git import GitUnmerge
from steps.step import CachingStep
logger = logging.getLogger(__name__)
@@ -33,19 +32,11 @@ def process_all_jobs(
for env in environemnts:
for step in env.pipeline:
try:
onupdate()
step.progress()
if (isinstance(step, GitUnmerge) or (
isinstance(step, CachingStep) and isinstance(step.step, GitUnmerge))) and len(
env.branches) == 0:
# TODO Move to separate job.
# If branches list empty, need to find any brunch which includes commit and add pair (branch, commit)
# If branches not empty, need to find most priority branch (project specific) and add (branch, HEAD)
env.branches = step.progress()
logger.info(f"Branches on startup resolved {env.id}, job {step.name}: {env.branches}")
except BaseException as e:
error_msg = f"Error processing release {env.id}, job {step.name}: {str(e)}"
logger.error(error_msg)
onupdate()
has_error = True
finally:
onupdate()
+119
View File
@@ -0,0 +1,119 @@
import asyncio
import json
import logging
from typing import Any, Awaitable, Callable, Dict, List, Optional, Iterator
import websockets
logger = logging.getLogger(__name__)
class SecondaryConnector:
def __init__(
self,
url: str,
on_update: Callable[[], Awaitable[None]],
) -> None:
self._url = url
self.branches: Dict[str, Any] = {}
self.environments: Dict[str, Any] = {}
self._task: Optional[asyncio.Task[None]] = None
self._on_update = on_update
self._ws: Optional[Any] = None
def start(self) -> None:
self._task = asyncio.create_task(self._connect())
@property
def is_active(self) -> bool:
return self._task is not None and not self._task.done()
async def send(self, msg: Dict[str, Any]) -> None:
payload = json.dumps(msg)
max_attempts = 3
timeout_seconds = 10.0
for attempt in range(1, max_attempts + 1):
if self._ws:
try:
await asyncio.wait_for(self._ws.send(payload), timeout=timeout_seconds)
return
except Exception as e:
logger.warning(f"Send attempt {attempt}/{max_attempts} failed for {self._url}: {e}")
if attempt < max_attempts:
await asyncio.sleep(timeout_seconds * attempt)
logger.error(f"Dropping message for secondary {self._url} after {max_attempts} attempts")
async def _connect(self) -> None:
logger.info(f"Connecting to secondary at {self._url}")
reconnect_delay_seconds = 5
while True:
try:
ws_url = (
self._url
.replace('http://', 'ws://')
.replace('https://', 'wss://')
+ '/ws'
)
async with websockets.connect(
ws_url,
ping_interval=20,
ping_timeout=20,
close_timeout=5,
) as ws:
self._ws = ws
logger.info(f"Connected to secondary WebSocket at {self._url}")
reconnect_delay_seconds = 5
async for msg in ws:
parsed = json.loads(msg)
if "branches" in parsed:
self.branches = parsed["branches"]
await self._on_update()
elif "environments" in parsed:
self.environments = parsed["environments"]
await self._on_update()
elif "error" in parsed:
await self._on_update()
except asyncio.CancelledError:
raise
except Exception:
logger.exception(
f"Unexpected secondary connector error for {self._url}. "
f"Retrying in {reconnect_delay_seconds}s"
)
finally:
self._ws = None
await asyncio.sleep(reconnect_delay_seconds)
reconnect_delay_seconds = min(reconnect_delay_seconds * 2, 60)
class SecondaryManager():
def __init__(self,
urls: str,
on_update: Callable[[], Awaitable[None]],
) -> None:
self.lst: List[SecondaryConnector] = []
urls_list = [it.strip() for it in urls.split(",") if it.strip()]
logger.info(f"SECONDARY_BRENCHER is {urls_list}")
for url in urls_list:
connector = SecondaryConnector(
url,
on_update=on_update,
)
connector.start()
self.lst.append(connector)
def __iter__(self) -> Iterator[SecondaryConnector]:
return iter(self.lst)
async def send(self, msg: Dict[str, Any]) -> None:
await asyncio.gather(*(connector.send(msg) for connector in self.lst), return_exceptions=True)
+53 -12
View File
@@ -15,7 +15,7 @@ from steps.git import GitClone, HasVersion
logger = logging.getLogger(__name__)
class DockerComposeBuild(AbstractStep[List[str]]):
class DockerComposeBuild(AbstractStep[Dict[str, str]]):
def __init__(self,
wd: GitClone, # TODO should be CheckoutMerged
docker_repo_username: str,
@@ -23,7 +23,9 @@ class DockerComposeBuild(AbstractStep[List[str]]):
docker_compose_path: str,
docker_repo_url: str,
publish: bool,
envs: Callable[[], Dict[str, Any]], **kwargs: Any) -> None:
envs: Callable[[], Dict[str, Any]],
build_cache: bool = False,
**kwargs: Any) -> None:
super().__init__(**kwargs)
self.wd = wd
self.envs = envs
@@ -32,12 +34,14 @@ class DockerComposeBuild(AbstractStep[List[str]]):
self.docker_compose_path = docker_compose_path
self.docker_repo_url = docker_repo_url
self.publish = publish
self.build_cache = build_cache
def progress(self) -> List[str]:
def progress(self) -> Dict[str, str]:
"""
Build and push Docker images defined in a docker-compose file.
Returns a dict mapping image name to its SHA digest.
"""
images_built = []
image_shas: Dict[str, str] = {}
try:
# Authenticate to docker repo
client = docker.DockerClient(base_url='unix://var/run/docker.sock')
@@ -57,38 +61,53 @@ class DockerComposeBuild(AbstractStep[List[str]]):
services = compose.get('services', {})
for name, svc in services.items():
if svc.get('build') is None:
build_section = svc.get('build')
if build_section is None:
continue
build_ctx = os.path.join(os.path.dirname(docker_compose_absolute_path), svc.get('build'))
if isinstance(build_section, dict):
build_ctx = build_section.get('context', '.')
build_dockerfile = build_section.get('dockerfile', 'Dockerfile')
else:
build_ctx = build_section
build_dockerfile = 'Dockerfile'
build_ctx = os.path.join(os.path.dirname(docker_compose_absolute_path), build_ctx)
build_dockerfile = os.path.join(os.path.dirname(docker_compose_absolute_path), build_dockerfile)
image = svc.get('image')
if not build_ctx or not image:
continue
if self.publish:
# Check if image exists in remote repo
try:
client.images.pull(image)
img = client.images.pull(image)
logger.info(f"Image {image} already exists in repo, skipping build.")
if img.id is not None:
image_shas[image] = img.id
continue
except Exception:
pass
else:
# Check if image exists locally
try:
client.images.get(image)
img = client.images.get(image)
logger.info(f"Image {image} already exists locally, skipping build.")
if img.id is not None:
image_shas[image] = img.id
continue
except docker_errors.ImageNotFound:
pass
logger.info(f"Building image {image} from {build_ctx}")
client.images.build(path=build_ctx, tag=image, nocache=True, rm=True)
logger.info(f"Building image {image} from {build_ctx}, {build_dockerfile}")
img, _ = client.images.build(path=build_ctx, dockerfile=build_dockerfile, tag=image, nocache=not self.build_cache, rm=True)
if self.publish:
logger.info(f"Pushing image {image}")
for line in client.images.push(image, stream=True, decode=True):
logger.debug(line)
images_built.append(image)
return images_built
if img.id is not None:
image_shas[image] = img.id
return image_shas
except Exception as e:
raise e
@@ -267,4 +286,26 @@ class DockerSwarmDeploy(AbstractStep[str]):
logger.error(f"Stack deploy failed: {result.stderr}")
raise RuntimeError(f"Stack deploy failed: {result.stderr}")
logger.info(f"Stack deployed successfully: {result.stdout}")
# Check health of all deployed services
client = docker.DockerClient(base_url='unix://var/run/docker.sock')
not_running = []
for svc_name in expected_services:
full_svc_name = f"{self.stack_name}_{svc_name}"
services_list = client.services.list(filters={"name": full_svc_name})
if not services_list:
not_running.append(f"{full_svc_name} (not found)")
continue
svc = services_list[0]
spec_mode = svc.attrs.get("Spec", {}).get("Mode", {})
desired_replicas = spec_mode.get("Replicated", {}).get("Replicas", 1)
running_tasks = [
t for t in svc.tasks()
if t.get("Status", {}).get("State") == "running" and t.get("DesiredState") == "running"
]
if len(running_tasks) < desired_replicas:
not_running.append(f"{full_svc_name} ({len(running_tasks)}/{desired_replicas} replicas running)")
if not_running:
raise BaseException(f"Services not started yet: {', '.join(not_running)}")
return f"Stack deployed successfully: {result.stdout}"
+4
View File
@@ -71,6 +71,10 @@ class DockerImageBuild(AbstractStep[DockerImageBuildResult]):
if len(existing_image) > 0 and not self.nocache:
logger.info(f"Image {full_image} already exists locally with ID: {existing_image}")
build_logs: List[str] = []
elif self.env.dry:
logger.info(f"DRY RUN: Would build image {full_image}")
raise BaseException(f"Dry run - skipping actual build for {full_image}")
else:
# Build the image
build_logs = []
+48 -25
View File
@@ -4,19 +4,23 @@ import os
import shutil
import tempfile
from dataclasses import dataclass
import traceback
from typing import List, Tuple, Set, Dict, Any, Mapping, runtime_checkable
import git
from enironment import AbstractStep
from enironment import AbstractStep, SharedState
from git.objects import Commit
logger = logging.getLogger(__name__)
class GitClone(AbstractStep[str]):
def __init__(self, repo_path: str | None = None, branchNamePrefix: str = "", credEnvPrefix: str = "GIT",
def __init__(self, url: str, repo_path: str | None = None,
branchNamePrefix: str = "", credEnvPrefix: str = "GIT",
**kwargs: Any):
super().__init__(**kwargs)
self.url = url
self.repo_path = repo_path
self.branchNamePrefix = branchNamePrefix
self.credEnvPrefix = credEnvPrefix
@@ -24,7 +28,7 @@ class GitClone(AbstractStep[str]):
def _get_auth_git_url(self, url: str) -> str:
username = os.getenv(f'{self.credEnvPrefix}_USERNAME')
password = os.getenv(f'{self.credEnvPrefix}_PASSWORD')
if username and password:
if username and password and '://' in url:
# Extract protocol and the rest of the URL
protocol, rest = url.split('://')
return f"{protocol}://{username}:{password}@{rest}"
@@ -32,32 +36,33 @@ class GitClone(AbstractStep[str]):
return url
def progress(self) -> str:
repo_url = self.url
self.repo_path = self.repo_path or os.path.join(tempfile.gettempdir(),
f"{self.env.id}_{hashlib.sha1(self.env.repo.encode()).hexdigest()[:5]}")
logger.info(f"Cloning repository {self.env.repo} to {self.repo_path}")
f"{self.env.id}_{hashlib.sha1(repo_url.encode()).hexdigest()[:5]}")
logger.info(f"Cloning repository {repo_url} to {self.repo_path}")
os.makedirs(self.repo_path, exist_ok=True)
try:
if os.path.exists(os.path.join(self.repo_path, ".git")):
logger.info(f"Repository already cloned at {self.repo_path}, fetching updates.")
repo = git.Repo(self.repo_path)
if repo.remotes.origin.url != self._get_auth_git_url(self.env.repo):
repo.remotes.origin.set_url(self._get_auth_git_url(self.env.repo))
if 'origin' not in repo.remotes or repo.remotes.origin.url != self._get_auth_git_url(repo_url):
repo.create_remote('origin', self._get_auth_git_url(repo_url))
result = repo.remotes.origin.fetch(prune=True)
if not result or any(fetch_info.flags & fetch_info.ERROR for fetch_info in result):
raise BaseException(f"Failed to fetch updates for {self.env.repo}")
raise BaseException(f"Failed to fetch updates for {repo_url}")
else:
repo = git.Repo.init(self.repo_path)
repo.remotes.append(repo.create_remote(
'origin',
self._get_auth_git_url(self.env.repo)
self._get_auth_git_url(repo_url)
))
if self.branchNamePrefix != "":
repo.config_writer().set_value('remote "origin"', "fetch",
f"+refs/heads/{self.branchNamePrefix}/*:refs/remotes/origin/{self.branchNamePrefix}/*").release()
repo.remotes.origin.fetch(prune=True)
if not os.path.exists(os.path.join(self.repo_path, ".git")):
raise BaseException(f"Failed to clone repository {self.env.repo} to {self.repo_path}")
raise BaseException(f"Failed to clone repository {repo_url} to {self.repo_path}")
except BaseException as e:
logger.error(f"Error during git clone/fetch, removing directory {self.repo_path}: {str(e)}")
shutil.rmtree(self.repo_path)
@@ -111,12 +116,14 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
wd: GitClone
def __init__(self, wd: GitClone,
desired_branches: AbstractStep[SharedState],
git_user_email: str,
git_user_name: str,
push: bool = True,
**kwargs: Any):
super().__init__(**kwargs)
self.wd = wd
self.desired_branches = desired_branches
self.git_user_email = git_user_email
self.git_user_name = git_user_name
self.push = push
@@ -136,16 +143,20 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
return visited
def find_desired_commits(self, repo: git.Repo) -> Dict[Commit, str]:
def find_desired_commits(self, repo: git.Repo, branches: List[Tuple[str, str]]) -> Dict[Commit, str]:
# Extract commit ids for the selected branches
commit_ids: Dict[Commit, str] = {}
for branch_pair in self.env.branches:
branch_name, desired_commit = branch_pair
if desired_commit == 'HEAD':
commit = repo.commit(f'origin/{branch_name}')
else:
commit = repo.commit(desired_commit)
commit_ids[commit] = branch_name
for branch_pair in branches:
try:
branch_name, desired_commit = branch_pair
if desired_commit == 'HEAD':
commit = repo.commit(f'origin/{branch_name}')
else:
commit = repo.commit(desired_commit)
commit_ids[commit] = branch_name
except BaseException as e:
stack = traceback.format_exception(type(e), e, e.__traceback__)
logger.error(f"Error finding desired commits for branch {branch_pair}: {str(e)}\n{''.join(stack)}")
return commit_ids
def progress(self) -> CheckoutAndMergeResult:
@@ -159,12 +170,13 @@ class CheckoutMerged(AbstractStep[CheckoutAndMergeResult]):
cw.set_value("user", "email", self.git_user_email)
cw.set_value("user", "name", self.git_user_name)
if len(self.env.branches) == 0:
desired = self.desired_branches.progress().branches
if len(desired) == 0:
raise BaseException(f"Empty branches set")
logger.info(f"Selected branches: {self.env.branches}")
logger.info(f"Selected branches: {desired}")
commit_ids = self.find_desired_commits(repo)
commit_ids = self.find_desired_commits(repo, desired)
logger.info(f"Commit ids for branches: {commit_ids}")
childs = _commits_childs(repo)
@@ -245,7 +257,13 @@ class HasVersion(Protocol):
version: str
class GitUnmerge(AbstractStep[List[Tuple[str, str]]]):
@dataclass
class GitUnmergeResult:
branches: List[Tuple[str, str]] # (branch_name, commit_hash) pairs
columns: Dict[str, Dict[str, str]] # column_name -> {branch_name: value}
class GitUnmerge(AbstractStep[GitUnmergeResult]):
wd: GitClone
def __init__(self, wd: GitClone,
@@ -255,7 +273,7 @@ class GitUnmerge(AbstractStep[List[Tuple[str, str]]]):
self.wd = wd
self.check = check
def progress(self) -> List[Tuple[str, str]]:
def progress(self) -> GitUnmergeResult:
wd = self.wd.progress()
deployState: Mapping[str, HasVersion] = self.check.progress()
@@ -282,7 +300,7 @@ class GitUnmerge(AbstractStep[List[Tuple[str, str]]]):
if len(branches) == 0 or branches[0].startswith('auto/'):
if childs is None:
childs = _commits_childs(repo)
commitsSet = set([commit])
commitsSet = {commit}
while len(commitsSet) > 0:
current = commitsSet.pop()
for child in childs.get(current, []):
@@ -300,6 +318,11 @@ class GitUnmerge(AbstractStep[List[Tuple[str, str]]]):
if len(commits) == 0:
raise BaseException(f"Unable to unmerge version: {version}")
return commits
deployed_col: Dict[str, str] = {b: c[:8] for b, c in commits}
return GitUnmergeResult(
branches=commits,
columns={"Deployed": deployed_col},
)
else:
raise BaseException(f"Version format not recognized: {version}")
+149
View File
@@ -0,0 +1,149 @@
import hashlib
import json
import logging
import os
import tempfile
import threading
import uuid
from dataclasses import replace
from typing import List, Tuple, Any
import git
from enironment import AbstractStep, SharedState, SharedStateHolder, SharedStateConflictError
from steps.git import GitUnmergeResult, GitClone
logger = logging.getLogger(__name__)
def _normalize_branches(branches: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
return sorted(branches, key=lambda x: (x[0], x[1]))
class SharedStateHolderInMemory(AbstractStep[SharedState], SharedStateHolder):
def __init__(self, unmerge: AbstractStep[GitUnmergeResult] | None, **kwargs: Any) -> None:
super().__init__(**kwargs)
self.state = SharedState([], False, token=uuid.uuid4().hex)
self.unmerge = unmerge
def set_branches(self, branches: List[Tuple[str, str]], expected_token: str | None = None) -> SharedState:
if expected_token is not None and expected_token != self.state.token:
raise SharedStateConflictError("State token mismatch")
self.state = replace(self.state, branches=_normalize_branches(branches), token=uuid.uuid4().hex)
return self.state
def set_dry(self, dry: bool, expected_token: str | None = None) -> SharedState:
if expected_token is not None and expected_token != self.state.token:
raise SharedStateConflictError("State token mismatch")
self.state = replace(self.state, dry=dry, token=uuid.uuid4().hex)
return self.state
def progress(self) -> SharedState:
if len(self.state.branches) == 0:
if self.unmerge:
unmerge = self.unmerge.progress()
self.state = replace(self.state, branches=unmerge.branches, token=uuid.uuid4().hex)
return self.state
class SharedStateHolderInGit(AbstractStep[SharedState], SharedStateHolder):
def __init__(
self,
wd: GitClone,
state_repo: GitClone,
state_branch: str = "state",
**kwargs: Any
) -> None:
super().__init__(**kwargs)
self.state_branch = state_branch
self.wd = wd
self.state_repo = state_repo
self._lock = threading.RLock()
def set_branches(self, branches: List[Tuple[str, str]], expected_token: str | None = None) -> SharedState:
with self._lock:
repo = self._ensure_repo(expected_token)
oldState = self._read(repo)
if expected_token is not None and expected_token != oldState.token:
raise SharedStateConflictError("State token mismatch")
newState = replace(oldState, branches=_normalize_branches(branches))
self._write(repo, newState)
return self._read(repo)
def set_dry(self, dry: bool, expected_token: str | None = None) -> SharedState:
with self._lock:
repo = self._ensure_repo(expected_token)
oldState = self._read(repo)
newState = replace(oldState, dry=dry)
self._write(repo, newState)
return self._read(repo)
def progress(self) -> SharedState:
with self._lock:
repo = self._ensure_repo(None)
return self._read(repo)
def file_path(self, repo: git.Repo) -> str:
return f"{repo.working_dir}/{self.env.id}.json"
def _ensure_repo(self, token: str | None = None) -> git.Repo:
repo_path = self.state_repo.progress()
repo = git.Repo(repo_path)
if token is None:
# Checkout state branch in detached HEAD state
token = f"origin/{self.state_branch}"
remote_refs = {ref.name for ref in repo.refs}
if not token in remote_refs:
raise BaseException("Branch not found")
repo.git.checkout("--detach", token)
return repo
def _read(self, repo: git.Repo) -> SharedState:
state_abs_path = self.file_path(repo)
os.makedirs(os.path.dirname(state_abs_path), exist_ok=True)
state_data: dict[str, Any] = {}
if os.path.exists(state_abs_path):
try:
with open(state_abs_path) as f:
state_data = json.load(f)
except json.JSONDecodeError as e:
raise Exception(f"Invalid JSON in state file {state_abs_path}: {str(e)}") from e
branches: List[Tuple[str, str]] = []
if isinstance(state_data.get("branches", []), list):
for item in state_data.get("branches", []):
if isinstance(item, (list, tuple)) and len(item) == 2:
branches.append((str(item[0]), str(item[1])))
dry = bool(state_data.get("dry", True))
return SharedState(branches=branches, dry=dry, token=repo.head.commit.hexsha)
def _write(self, repo: git.Repo, state: SharedState) -> None:
state_abs_path = self.file_path(repo)
os.makedirs(os.path.dirname(state_abs_path), exist_ok=True)
state_data = {
"branches": state.branches,
"dry": state.dry,
}
new_state_content = json.dumps(state_data, sort_keys=True, indent=2) + "\n"
with open(state_abs_path, "w") as f:
f.write(new_state_content)
repo.index.add([state_abs_path])
repo.index.commit(f"Update state for {self.env.id}")
try:
remote_state_ref = f"origin/{self.state_branch}"
repo.git.merge(remote_state_ref, no_edit=True)
repo.git.push("origin", f"HEAD:refs/heads/{self.state_branch}")
except git.GitCommandError as e:
try:
repo.git.merge("--abort")
except git.GitCommandError:
pass
raise SharedStateConflictError("State token mismatch")
+37 -10
View File
@@ -1,4 +1,6 @@
from typing import TypeVar, Generic
import hashlib
import json
from typing import TypeVar, Generic, Any
from enironment import AbstractStep
@@ -10,26 +12,51 @@ class NotReadyException(BaseException):
super().__init__(message)
def _stable_hash(obj: Any) -> str:
"""Compute a stable SHA-256 hash of an object via JSON serialization."""
try:
serialized = json.dumps(obj, sort_keys=True, default=repr)
except Exception:
serialized = repr(obj)
return hashlib.sha256(serialized.encode()).hexdigest()
class CachingStep(AbstractStep[T], Generic[T]):
_result: T | BaseException
reset_cache: bool = False
_input_hash: str | None
def __init__(self, step: AbstractStep[T]) -> None:
super().__init__(n=step.name)
self.step = step
self._result = NotReadyException(f"No result yet for {self.step.name}")
self._step = step
self._result = NotReadyException(f"No result yet for {self._step.name}")
self._input_hash = None
def _compute_input_hash(self) -> str:
"""Compute a stable hash of all AbstractStep dependency outputs."""
inputs: dict[str, Any] = {}
for attr_name, attr_value in vars(self._step).items():
if isinstance(attr_value, AbstractStep):
try:
inputs[attr_name] = attr_value.progress()
except BaseException as e:
inputs[attr_name] = repr(e)
return _stable_hash(inputs)
def progress(self) -> T:
if self.reset_cache or self._result is None or isinstance(self._result, BaseException):
current_hash = self._compute_input_hash()
if self._input_hash != current_hash or isinstance(self._result, BaseException):
try:
self._result = self.step.progress()
self._result = self._step.progress()
except BaseException as e:
self._result = e
self.reset_cache = False
self._input_hash = current_hash
if isinstance(self._result, BaseException):
raise self._result from None
else:
return self._result
return self._result
def reset(self) -> None:
self.reset_cache = True
self._input_hash = None
def __getattr__(self, name: str) -> Any:
"""Delegate unknown attributes to the wrapped step."""
return getattr(self._step, name)
+31
View File
@@ -0,0 +1,31 @@
import json
from dataclasses import asdict
import os
from typing import Any
class DataclassJSONEncoder(json.JSONEncoder):
def default(self, o: Any) -> Any:
if hasattr(o, '__dataclass_fields__'):
return asdict(o)
if isinstance(o, BaseException):
return str(o)
try:
return super().default(o)
except TypeError:
return str(o)
def custom_json_dumps(obj: Any) -> str:
return json.dumps(obj, cls=DataclassJSONEncoder)
def sigchld_handler(signum: int, frame: Any) -> None:
"""Reap zombie processes."""
while True:
try:
pid, status = os.waitpid(-1, os.WNOHANG)
if pid == 0:
break
except ChildProcessError:
break
+240
View File
@@ -0,0 +1,240 @@
import asyncio
import json
import logging
import os
import traceback
from contextlib import asynccontextmanager
from typing import Any, AsyncIterator, Dict, List, Optional, TypeVar
from fastapi import FastAPI, Response, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse
from app import App
from utils import custom_json_dumps
from secondary import SecondaryManager
logger = logging.getLogger(__name__)
FRONTEND_DIR = os.path.join(os.path.dirname(__file__), '../frontend')
T = TypeVar('T')
def _merge_dicts(a: Dict[str, T], b: Dict[str, T]) -> Dict[str, T]:
result: Dict[str, T] = {}
for k in (a.keys() | b.keys()):
if k not in a or k not in b:
result[k] = a.get(k, b.get(k)) # type: ignore[assignment]
continue
if (
isinstance(a[k], dict)
and isinstance(b[k], dict)
):
result[k] = _merge_dicts(a[k], b[k]) # type: ignore[arg-type,assignment]
else:
result[k] = b[k]
return result
class WebApp:
def __init__(self, core: App, port: int) -> None:
self.core = core
self.port = port
self.ws_connections: Dict[WebSocket, Dict[str, Any]] = {}
self._event_loop: Optional[asyncio.AbstractEventLoop] = None
self._server: Any = None
self.secondaryManager: Optional[SecondaryManager] = None
@asynccontextmanager
async def lifespan(fastapi_app: FastAPI) -> AsyncIterator[None]:
self._event_loop = asyncio.get_running_loop()
self.secondaryManager = SecondaryManager(
urls=os.getenv("SECONDARY_BRENCHER", ""),
on_update=lambda: self.broadcast_all(),
)
yield
self.app = FastAPI(lifespan=lifespan)
self.app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
self.app.get("/")(self.serve_index)
self.app.get("/state")(self.serve_state)
self.app.get("/branches")(self.serve_branches_route)
self.app.get("/{path:path}")(self.serve_static)
self.app.websocket("/ws")(self.websocket_endpoint)
# --- State assembly (local + secondary) ---
def get_global_envs_to_emit(self) -> Any:
local_envs = self.core.get_local_envs_to_emit()
merged: Dict[str, Any] = local_envs
for connector in self.secondaryManager or []:
v = connector.environments
common_keys = set(merged.keys()) & set(v.keys())
if len(common_keys) > 0:
self._schedule_async(self.broadcast_error({'message': f"Conflict: both master and secondary have environment with id {common_keys}"}))
merged = _merge_dicts(merged, v)
return merged
def get_global_branches_to_emit(self) -> Dict[str, Dict[str, List[Any]]]:
local_branches: Dict[str, Dict[str, List[Any]]] = self.core.get_local_branches_to_emit()
merged: Dict[str, Dict[str, List[Any]]] = local_branches
for connector in self.secondaryManager or []:
merged = _merge_dicts(merged, connector.branches)
return merged
# --- Async scheduling from sync callbacks ---
def _schedule_async(self, coro: Any) -> None:
if self._event_loop is None or not self._event_loop.is_running():
logger.warning("Dropping scheduled coroutine because the web event loop is not running")
coro.close()
return
try:
running_loop = asyncio.get_running_loop()
except RuntimeError:
running_loop = None
if running_loop is self._event_loop:
self._event_loop.create_task(coro)
else:
asyncio.run_coroutine_threadsafe(coro, self._event_loop)
# --- Broadcasting ---
async def broadcast(self, event: str, data: Any) -> None:
"""Broadcast a message to all connected WebSocket clients that have not seen this data yet."""
disconnected = set()
message = custom_json_dumps({event: data})
for websocket in self.ws_connections.keys():
if self.ws_connections[websocket].get(event) == message:
continue
try:
await websocket.send_text(message)
self.ws_connections[websocket][event] = message
except Exception as e:
logger.error(f"Error sending to websocket: {e}")
disconnected.add(websocket)
for ws in disconnected:
self.ws_connections.pop(ws, None)
async def broadcast_all(self) -> None:
await self.broadcast("branches", self.get_global_branches_to_emit())
await self.broadcast("environments", self.get_global_envs_to_emit())
async def broadcast_branches(self, data: Any) -> None:
await self.broadcast("branches", data)
async def broadcast_environments(self, data: Any) -> None:
await self.broadcast("environments", data)
async def broadcast_error(self, data: Any) -> None:
await self.broadcast("error", data)
def emit_envs(self) -> None:
"""Sync callback used by App's processing loop to push updates to clients."""
self._schedule_async(self.broadcast_branches(self.get_global_branches_to_emit()))
self._schedule_async(self.broadcast_environments(self.get_global_envs_to_emit()))
# --- Routes ---
async def serve_index(self) -> FileResponse:
return FileResponse(os.path.join(FRONTEND_DIR, 'index.html'))
async def serve_state(self) -> Response:
return Response(content=custom_json_dumps(self.get_global_envs_to_emit()), media_type="application/json")
async def serve_branches_route(self) -> Response:
return Response(content=custom_json_dumps(self.get_global_branches_to_emit()), media_type="application/json")
async def serve_static(self, path: str) -> FileResponse:
file_path = os.path.join(FRONTEND_DIR, path)
if os.path.exists(file_path) and os.path.isfile(file_path):
return FileResponse(file_path)
return FileResponse(os.path.join(FRONTEND_DIR, 'index.html'))
async def websocket_endpoint(self, websocket: WebSocket) -> None:
await websocket.accept()
self.ws_connections[websocket] = {}
try:
# Send initial state on connect and record it so duplicate broadcasts are suppressed
branches_payload = custom_json_dumps({"branches": self.get_global_branches_to_emit()})
envs_payload = custom_json_dumps({"environments": self.get_global_envs_to_emit()})
await websocket.send_text(branches_payload)
self.ws_connections[websocket]["branches"] = branches_payload
await websocket.send_text(envs_payload)
self.ws_connections[websocket]["environments"] = envs_payload
while True:
data = await websocket.receive_text()
message = json.loads(data)
if "update" in message:
update_data = message.get("update") or {}
logger.info(f"Received environment update: {update_data}")
if self.secondaryManager:
await self.secondaryManager.send({"update": update_data})
id = update_data.get('id', '')
if id == '':
self.core.request_reset()
elif id not in self.core.environments.keys() and id not in {j for it in self.secondaryManager or [] for j in it.environments.keys()}:
logger.warning(f"Received update for unknown environment id {id}")
continue
elif id in self.core.environments.keys():
env = self.core.environments.get(id, None)
expected_token = update_data.get('token', '')
if 'branches' in update_data:
if not env:
raise RuntimeError(f"Unknown env {update_data.get('id', '')}")
env.state.set_branches(update_data.get('branches', []), expected_token=expected_token)
if 'dry' in update_data:
if not env:
raise RuntimeError(f"Unknown env {update_data.get('id', '')}")
env.state.set_dry(bool(update_data['dry']), expected_token)
if env:
self.core.request_reset()
logger.info(f"Updated environment {env.id} branches to {update_data.get('branches')}, dry={update_data.get('dry')}")
self.core.notify_environment_update()
await self.broadcast_environments(self.get_global_envs_to_emit())
except WebSocketDisconnect:
self.ws_connections.pop(websocket, None)
except Exception as e:
logger.error(f"WebSocket error: {e}:{traceback.format_exc()}")
await self.broadcast_error({'message': f'{e}:{traceback.format_exc()}'})
# --- Run ---
async def start_async(self) -> None:
import uvicorn
config = uvicorn.Config(self.app, host='0.0.0.0', port=self.port, log_level='info')
self._server = uvicorn.Server(config)
try:
await self._server.serve()
except asyncio.CancelledError:
self._server.should_exit = True
raise
def start(self) -> None:
asyncio.run(self.start_async())
def stop(self) -> None:
if self._server is None:
return
if self._event_loop is not None and self._event_loop.is_running():
self._event_loop.call_soon_threadsafe(lambda: setattr(self._server, "should_exit", True))
else:
self._server.should_exit = True
View File
+8 -3
View File
@@ -16,13 +16,18 @@ services:
- PYTHONUNBUFFERED=1
- VIRTUAL_HOST=brencher.rudolf.keenetic.link
- LETSENCRYPT_HOST=brencher.rudolf.keenetic.link
- PROFILES=brencher2,torrserv_proxy,immich,registry
- SLAVE_BRENCHER=http://host.docker.internal:5002
# - SLAVE_BRENCHER='http://100.70.193.97:5002' .venv/bin/python3 backend/app.py brencher_localX
- PROFILES=brencher2,torrserv_proxy,immich,registry,gmail_mcp
- SECONDARY_BRENCHER=http://host.docker.internal:5002
deploy:
replicas: 1
restart_policy:
condition: on-failure
placement:
max_replicas_per_node: 1
constraints:
- node.hostname==odroid
# - node.hostname==rudolf-T
mode: replicated
networks:
- default
- brencher-net
+176 -99
View File
@@ -2,10 +2,8 @@
// You can add your Socket.IO and UI code here
console.log('app.js loaded');
// WebSocket namespaces
const WS_BRANCHES = '/ws/branches';
const WS_ENVIRONMENT = '/ws/environment';
const WS_ERRORS = '/ws/errors';
// WebSocket endpoint
const WS_URL = '/ws';
// DOM refs
const branchesList = document.getElementById('branches-list');
@@ -32,13 +30,17 @@ let environmentsRaw = [];
const selectedBranchesByEnv = {};
// serverSelectedBranchesByEnv mirrors server's last known selection for diffing
const serverSelectedBranchesByEnv = {};
// deployedCommitsByEnv: envId -> { branchName: deployedShortHash }
const deployedCommitsByEnv = {};
// serverTokenByEnv tracks current optimistic concurrency token per environment
const serverTokenByEnv = {};
// columnsByEnv: envId -> { columnName: { branchName: value } }
// Populated from any step result that contains a `columns` key (e.g. GitUnmerge)
const columnsByEnv = {};
// Socket instances
let wsBranches = null;
let wsEnv = null;
let wsErr = null;
// Per-environment dry run state: envId -> bool
const dryRunByEnv = {};
// Single WebSocket connection
let ws = null;
function showStatus(message, isError = false) {
statusMessage.textContent = message;
@@ -49,14 +51,16 @@ function showStatus(message, isError = false) {
closeStatus.onclick = () => statusBar.classList.add('hidden');
function checkForPendingChanges() {
function isChangesPending() {
// Determine if any environment has changed selections
const changed = Object.keys(selectedBranchesByEnv).some(envId => {
return Object.keys(selectedBranchesByEnv).some(envId => {
const localSel = [...(selectedBranchesByEnv[envId] || [])].sort();
const serverSel = [...(serverSelectedBranchesByEnv[envId] || [])].sort();
return JSON.stringify(localSel) !== JSON.stringify(serverSel);
});
if (changed) applyChangesBtn.classList.remove('hidden');
}
function checkForPendingChanges() {
if (isChangesPending()) applyChangesBtn.classList.remove('hidden');
else applyChangesBtn.classList.add('hidden');
}
@@ -64,9 +68,10 @@ function filterBranches() {
const filterText = branchFilter.value.toLowerCase().trim();
filteredBranches = branches.filter(({envId, branch, commits}) => {
const selList = selectedBranchesByEnv[envId] || [];
const deployedMap = deployedCommitsByEnv[envId] || {};
const isSelected = selList.some(([b]) => b === branch);
const isDeployed = deployedMap[branch] && deployedMap[branch] !== 'N/A';
// Always show branches that appear in any column (e.g. deployed branches from GitUnmerge)
const envCols = columnsByEnv[envId] || {};
const isInColumns = Object.values(envCols).some(colData => branch in colData);
const isFiltered = !!filterText && branch.toLowerCase().includes(filterText);
const isFilteredByCommit = !!filterText && Array.isArray(commits) && commits.some(c => {
const shortHash = c.hexsha ? c.hexsha.substring(0, 8).toLowerCase() : '';
@@ -75,7 +80,7 @@ function filterBranches() {
(c.message && c.message.toLowerCase().includes(filterText)) ||
(c.author && c.author.toLowerCase().includes(filterText));
});
return showAllBranches || isSelected || isDeployed || isFiltered || isFilteredByCommit;
return showAllBranches || isSelected || isInColumns || isFiltered || isFilteredByCommit;
});
renderBranches();
}
@@ -88,12 +93,14 @@ function getCommitDropdownOptions(envId, branchName, desiredCommit) {
if (commits.length) {
const head = commits[0];
const headShort = head.hexsha ? head.hexsha.substring(0, 8) : '';
options += `<option value="HEAD" ${desiredCommit === 'HEAD' ? 'selected' : ''}>HEAD (${headShort} - ${(head.author || '').substring(0, 25)} - ${head.message ? head.message.substring(0, 50) + (head.message.length > 50 ? '...' : '') : ''})</option>`;
const headMsg = escapeHtml(head.message || '');
options += `<option value="HEAD" ${desiredCommit === 'HEAD' ? 'selected' : ''}>HEAD (${headShort} - ${(head.author || '').substring(0, 25)} - ${headMsg ? headMsg.substring(0, 50) + (headMsg.length > 50 ? '...' : '') : ''})</option>`;
commits.forEach(c => {
const full = c.hexsha;
const shortHash = full ? full.substring(0, 8) : '';
const isSelected = desiredCommit === full || desiredCommit === shortHash;
options += `<option value="${full}" ${isSelected ? 'selected' : ''}>${shortHash} - ${(c.author || '').substring(0, 25)} - ${c.message ? c.message.substring(0, 50) + (c.message.length > 50 ? '...' : '') : ''}</option>`;
const cMsg = escapeHtml(c.message || '');
options += `<option value="${full}" ${isSelected ? 'selected' : ''}>${shortHash} - ${(c.author || '').substring(0, 25)} - ${cMsg ? cMsg.substring(0, 50) + (cMsg.length > 50 ? '...' : '') : ''}</option>`;
});
} else {
options += `<option value="HEAD" ${desiredCommit === 'HEAD' ? 'selected' : ''}>HEAD (no commits)</option>`;
@@ -120,10 +127,19 @@ function branchHasCommit(envId, branchName, commit) {
}) || false;
}
function escapeHtml(str) {
return String(str)
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#39;');
}
let renderedBranches = []
function renderBranches() {
if (JSON.stringify(filteredBranches) === JSON.stringify(renderedBranches)) {
if (JSON.stringify(filteredBranches) === JSON.stringify(renderedBranches) && JSON.stringify(dryRunByEnv) === JSON.stringify(renderedDryRunState)) {
return
}
checkForPendingChanges();
@@ -142,16 +158,23 @@ function renderBranches() {
}, {});
branchesList.innerHTML = Object.entries(groups).map(([envId, {envName, rows}]) => {
const envCols = columnsByEnv[envId] || {};
const colNames = Object.keys(envCols);
return `
<div class="env-block">
<h3 class="env-title">Environment: ${envName || envId}</h3>
<h3 class="env-title">
Environment: ${envName || envId}
<button class="dry-run-btn ${dryRunByEnv[envId] ? 'dry-run-active' : ''}" data-env="${envId}" title="${dryRunByEnv[envId] ? 'Dry run on — click to resume' : 'Running — click to pause (dry run)'}">
${dryRunByEnv[envId] ? '▶' : '⏸'}
</button>
</h3>
<table class="branches-table">
<thead>
<tr>
<th>Deploy</th>
<th>Branch</th>
<th>Desired Commit</th>
<th>Deployed</th>
${colNames.map(n => `<th>${escapeHtml(n)}</th>`).join('')}
</tr>
</thead>
<tbody>
@@ -159,8 +182,6 @@ function renderBranches() {
const selList = selectedBranchesByEnv[eId] || [];
const pair = selList.find(([b]) => b === branch);
const desiredCommit = pair ? pair[1] : 'HEAD';
const deployedMap = deployedCommitsByEnv[eId] || {};
const deployedCommit = deployedMap[branch] || 'N/A';
const hasCommit = branchHasCommit(eId, branch, desiredCommit);
return `
<tr class="branch-row" data-env="${eId}" data-branch="${branch}">
@@ -174,7 +195,7 @@ function renderBranches() {
value="${desiredCommit !== 'HEAD' && !hasCommit ? desiredCommit : ''}"
placeholder="Enter commit ID" style="display: ${desiredCommit !== 'HEAD' && !hasCommit ? 'inline-block' : 'none'};" />
</td>
<td>${deployedCommit}</td>
${colNames.map(n => `<td>${escapeHtml((envCols[n] && envCols[n][branch]) || '')}</td>`).join('')}
</tr>`;
}).join('')}
</tbody>
@@ -222,7 +243,17 @@ function renderBranches() {
updateBranchCommit(envId, branch, commitId || 'HEAD');
};
});
branchesList.querySelectorAll('.dry-run-btn').forEach(btn => {
btn.onclick = e => {
const envId = e.currentTarget.dataset.env;
if (ws && ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify({ update: { id: envId, dry: !dryRunByEnv[envId], token: serverTokenByEnv[envId] } }));
}
filterBranches();
};
});
renderedBranches = filteredBranches;
renderedDryRunState = {...dryRunByEnv};
}
function updateBranchCommit(envId, branch, commit) {
@@ -239,7 +270,10 @@ function renderJobs() {
jobsList.innerHTML = '<p class="loading">No jobs found.</p>';
return;
}
jobsList.innerHTML = environmentsRaw.map((envObj) => {
jobsList.innerHTML = environmentsRaw
.slice()
.sort((a, b) => (a.name || a.id || '').localeCompare(b.name || b.id || ''))
.map((envObj) => {
var linksHtml = ""
const jobsArr = envObj.pipeline || [];
if (Array.isArray(jobsArr)) {
@@ -279,15 +313,18 @@ function renderJobs() {
const storageKey = 'jobSpoiler:' + key;
const safeId = 'spoiler-' + encodeURIComponent(key).replace(/[^a-zA-Z0-9_-]/g, '_');
const isError = job.error;
const isRunning = !!job.is_running;
const openByDefault = isError || (window._jobSpoilerState && window._jobSpoilerState[storageKey] === 'open');
return `
<div class="job-item">
<div class="job-header" style="cursor:pointer;font-weight:bold;"
onclick="toggleJobSpoiler('${key}', '${safeId}')">
${isError
? `<span style="color:#dc3545;font-weight:bold;margin-right:6px;" title="Error">!</span>`
: `<span style="color:#28a745;font-weight:bold;margin-right:6px;" title="OK">✔</span>`}
${isRunning
? `<span class="step-spinner" title="Running"></span>`
: isError
? `<span style="color:#dc3545;font-weight:bold;margin-right:6px;" title="Error">!</span>`
: `<span style="color:#28a745;font-weight:bold;margin-right:6px;" title="OK">✔</span>`}
${envObj.id} - ${job.name}
</div>
<div id="${safeId}" class="job-spoiler" style="display: ${openByDefault ? 'block' : 'none'}; margin-top:8px;">
@@ -319,7 +356,9 @@ toggleJobSpoiler = function (key, safeId) {
}
};
refreshBranchesBtn.onclick = () => {
wsEnv.emit('update', {id: ""});
if (ws && ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify({ update: { id: "" } }));
}
showStatus('Refreshing...');
};
@@ -343,78 +382,117 @@ applyChangesBtn.onclick = () => {
updateEnvironment();
};
// Socket.IO setup
function setupSocketIO() {
wsBranches = io(WS_BRANCHES);
wsEnv = io(WS_ENVIRONMENT);
wsErr = io(WS_ERRORS);
// Single WebSocket setup
function setupWebSockets() {
// Close existing connection before reconnecting
if (ws && ws.readyState !== WebSocket.CLOSED) {
ws.onclose = null; // Prevent recursive reconnect
ws.close();
}
wsBranches.on('branches', data => {
// data: { envId: { branchName: commits[] } }
branches = Object.entries(data).flatMap(([envId, branchMap]) =>
Object.entries(branchMap).map(([branchName, commitList]) => {
const envObj = (environmentsRaw.find((e) => e.id === envId) || [null])[0];
return {
envId,
envName: envObj ? (envObj.name || envObj.id) : envId,
branch: branchName,
commits: commitList
};
})
);
filterBranches();
showStatus('Branches updated.');
});
const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
const host = window.location.host;
wsEnv.on('environments', data => {
payload = data || {};
environmentsRaw = Object.values(payload);
ws = new WebSocket(`${protocol}//${host}${WS_URL}`);
ws.onopen = () => console.log('WebSocket connected');
ws.onmessage = (event) => {
try {
const message = JSON.parse(event.data);
if ('branches' in message) {
const data = message.branches;
// data: { envId: { branchName: commits[] } }
branches = Object.entries(data).flatMap(([envId, branchMap]) =>
Object.entries(branchMap).map(([branchName, commitList]) => {
const envObj = (environmentsRaw.find((e) => e.id === envId) || [null])[0];
return { envId, envName: envObj ? (envObj.name || envObj.id) : envId, branch: branchName, commits: commitList };
})
);
filterBranches();
showStatus('Branches updated.');
} else if ('environments' in message) {
const data = message.environments;
payload = data || {};
environmentsRaw = Object.values(payload);
// Sync selections and deployed commits per environment
environmentsRaw.forEach((envObj) => {
if (!envObj) return;
const envId = envObj.id || envObj.name || 'unknown';
// Branch selections
if (Array.isArray(envObj.branches) && envObj.branches.length > 0) {
if (Array.isArray(envObj.branches[0])) {
selectedBranchesByEnv[envId] = [...envObj.branches];
} else {
selectedBranchesByEnv[envId] = envObj.branches.map(b => [b, 'HEAD']);
}
serverSelectedBranchesByEnv[envId] = [...selectedBranchesByEnv[envId]];
} else if (!selectedBranchesByEnv[envId]) {
selectedBranchesByEnv[envId] = [];
serverSelectedBranchesByEnv[envId] = [];
}
const jobsArr = envObj.pipeline || []
// Deployed commits from GitUnmerge job
const gitUnmergeJob = Array.isArray(jobsArr) ? jobsArr.find(j => j.name === 'GitUnmerge') : null;
deployedCommitsByEnv[envId] = {};
if (gitUnmergeJob && Array.isArray(gitUnmergeJob.status)) {
gitUnmergeJob.status.forEach(([commitHash, branchNames]) => {
if (Array.isArray(branchNames)) {
branchNames.forEach(bName => {
deployedCommitsByEnv[envId][bName] = commitHash.substring(0, 8);
// Sync selections and column data per environment
environmentsRaw.forEach((envObj) => {
if (!envObj) return;
const envId = envObj.id || envObj.name || 'unknown';
// Server state is the source of truth for dry-run rendering.
dryRunByEnv[envId] = !!envObj.dry;
if (typeof envObj.branches_token === 'string') {
serverTokenByEnv[envId] = envObj.branches_token;
}
if (!isChangesPending()) {
if (Array.isArray(envObj.branches) && envObj.branches.length > 0) {
if (Array.isArray(envObj.branches[0])) {
selectedBranchesByEnv[envId] = [...envObj.branches];
} else {
selectedBranchesByEnv[envId] = envObj.branches.map(b => [b, 'HEAD']);
}
serverSelectedBranchesByEnv[envId] = [...selectedBranchesByEnv[envId]];
} else if (!selectedBranchesByEnv[envId]) {
selectedBranchesByEnv[envId] = [];
serverSelectedBranchesByEnv[envId] = [];
}
}
// Extract columns from all job statuses
columnsByEnv[envId] = {};
const jobsArr = envObj.pipeline || [];
if (Array.isArray(jobsArr)) {
jobsArr.forEach(job => {
if (job.status == null || job.error) return;
const status = job.status;
// New structured format: status is an object with a `columns` key
if (typeof status === 'object' && !Array.isArray(status) && status.columns &&
typeof status.columns === 'object') {
Object.entries(status.columns).forEach(([colName, colData]) => {
if (colData && typeof colData === 'object' && !Array.isArray(colData)) {
columnsByEnv[envId][colName] = {...colData};
}
});
}
});
}
});
// Rebuild branches list environment names for rows already present
branches = branches.map(b => {
const envEntry = environmentsRaw.find((e) => (e.id || e.name) === b.envId);
return { ...b, envName: envEntry ? (envEntry.name || envEntry.id) : b.envName };
});
filterBranches();
renderJobs();
showStatus('Environments updated.');
} else if ('error' in message) {
console.error('Error from server:', message.error);
if (message.error && message.error.code === 'BRANCH_STATE_CONFLICT' && message.error.envId) {
const envId = message.error.envId;
const current = message.error.current_state || {};
if (Array.isArray(current.branches)) {
selectedBranchesByEnv[envId] = [...current.branches];
serverSelectedBranchesByEnv[envId] = [...current.branches];
}
if (typeof current.token === 'string') {
serverTokenByEnv[envId] = current.token;
}
filterBranches();
}
showStatus(message.error.message || 'Unknown error', true);
}
});
// Rebuild branches list environment names for rows already present
branches = branches.map(b => {
const envEntry = environmentsRaw.find((e) => (e.id || e.name) === b.envId);
return {...b, envName: envEntry ? (envEntry.name || envEntry.id) : b.envName};
});
filterBranches();
renderJobs();
showStatus('Environments updated.');
});
wsEnv.on('disconnect', () => showStatus('Disconnected from environment websocket.', true));
wsBranches.on('disconnect', () => showStatus('Disconnected from branches websocket.', true));
wsErr.on('error', data => showStatus(data.message || 'Unknown error', true));
checkForPendingChanges();
} catch (e) {
console.error('Error processing WebSocket message:', e);
}
};
ws.onclose = () => {
showStatus('Disconnected from server.', true);
setTimeout(() => setupWebSockets(), 5000);
};
ws.onerror = (error) => {
console.error('WebSocket error:', error);
};
}
function updateEnvironment() {
@@ -423,18 +501,17 @@ function updateEnvironment() {
const localSel = [...(selectedBranchesByEnv[envId] || [])].sort();
const serverSel = [...(serverSelectedBranchesByEnv[envId] || [])].sort();
if (JSON.stringify(localSel) !== JSON.stringify(serverSel)) {
wsEnv.emit('update', {id: envId, branches: selectedBranchesByEnv[envId]});
// Optimistically sync server state
serverSelectedBranchesByEnv[envId] = [...selectedBranchesByEnv[envId]];
if (ws && ws.readyState === WebSocket.OPEN) {
ws.send(JSON.stringify({ update: { id: envId, branches: selectedBranchesByEnv[envId], token: serverTokenByEnv[envId] } }));
}
branchFilter.value = '';
}
});
checkForPendingChanges();
showStatus('Changes applied (pending server confirmation).');
}
// Initial load
setupSocketIO();
setupWebSockets();
showStatus('Loading branches...');
renderBranches();
renderJobs();
-1
View File
@@ -47,7 +47,6 @@
</main>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.6.1/socket.io.js"></script>
<script src="app.js"></script>
</body>
</html>
+47
View File
@@ -193,3 +193,50 @@ input[type="checkbox"] {
margin: 0;
transform: scale(1.2);
}
/* Dry run play/pause button */
.env-title {
display: flex;
align-items: center;
gap: 12px;
}
.dry-run-btn {
margin-left: auto;
background: none;
border: none;
cursor: pointer;
font-size: 1.2rem;
line-height: 1;
padding: 2px 6px;
border-radius: 4px;
color: #aaaaaa;
transition: color 0.15s, background 0.15s;
}
.dry-run-btn:hover {
background: rgba(0,0,0,0.06);
}
.dry-run-btn.dry-run-active {
color: #ffc107;
}
/* Step spinner */
@keyframes step-spin {
from { transform: rotate(0deg); }
to { transform: rotate(360deg); }
}
.step-spinner {
display: inline-block;
width: 12px;
height: 12px;
border: 2px solid #007bff;
border-top-color: transparent;
border-radius: 50%;
animation: step-spin 0.7s linear infinite;
margin-right: 6px;
vertical-align: middle;
flex-shrink: 0;
}
+5 -5
View File
@@ -45,19 +45,19 @@ disallow_any_unimported = False
[mypy-docker.*]
ignore_missing_imports = True
[mypy-flask_socketio.*]
ignore_missing_imports = True
[mypy-dotenv.*]
ignore_missing_imports = True
[mypy-yaml.*]
ignore_missing_imports = True
[mypy-flask.*]
[mypy-websockets.*]
ignore_missing_imports = True
[mypy-socketio.*]
[mypy-uvicorn.*]
ignore_missing_imports = True
[mypy-fastapi.*]
ignore_missing_imports = True
[mypy-pytest.*]
+13
View File
@@ -0,0 +1,13 @@
-r requirements.txt
pytest
pytest-asyncio
playwright
pytest-playwright
mypy
flake8
ty
types-PyYAML
types-requests
types-docker
testcontainers
+3 -13
View File
@@ -1,17 +1,7 @@
flask
flask-socketio
fastapi
uvicorn[standard]
websockets
gitpython
python-dotenv
docker
PyYAML
websocket-client
pytest
pytest-asyncio
mypy
ty
types-PyYAML
types-requests
types-flask-socketio
types-docker
testcontainers
+74
View File
@@ -0,0 +1,74 @@
import subprocess
from pathlib import Path
from enironment import Environment
from steps.docker_plain import DockerImageBuild, DockerContainerCheck, DockerContainerDeploy
from steps.git import GitClone, CheckoutMerged, GitUnmerge
from steps.shared_state import SharedStateHolderInMemory
REPO_ROOT = Path(__file__).resolve().parents[2]
def _git_output(*args: str) -> str:
try:
return subprocess.check_output(
["git", "-C", str(REPO_ROOT), *args],
text=True,
stderr=subprocess.STDOUT,
).strip()
except subprocess.CalledProcessError as exc:
raise RuntimeError(f"Unable to read git metadata for nginx test config: {' '.join(args)}\n{exc.output}") from exc
CURRENT_BRANCH = _git_output("rev-parse", "--abbrev-ref", "HEAD")
CURRENT_COMMIT = _git_output("rev-parse", "HEAD")
clone = GitClone(url="https://github.com/rudolf1/brencher.git")
container_check = DockerContainerCheck(
container_name="test_plain-container"
)
unmerge = GitUnmerge(wd=clone, check=container_check)
state = SharedStateHolderInMemory(unmerge=unmerge)
checkoutMerged = CheckoutMerged(clone,
desired_branches=state,
push=False,
git_user_email="rudolfss13@gmail.com",
git_user_name="test_bot"
)
# Step 1: Build the image
image_build = DockerImageBuild(
wd=checkoutMerged,
dockerfile_path="Dockerfile_test_app",
image_name="test_plain",
image_tag=lambda: "auto-" + checkoutMerged.progress().version,
nocache=False,
)
# Step 3: Deploy the container
container_deploy = DockerContainerDeploy(
image_build=image_build,
container_name="test_plain-container",
ports={"5001/tcp": 5002},
environment={"PROFILES": "no_profiles"},
restart_policy={"Name": "unless-stopped"},
)
__all__ = ["test_local1"]
test_local1 = Environment(
id="test_local1",
state=state,
pipeline=[
clone,
state,
checkoutMerged,
image_build,
container_deploy,
container_deploy,
unmerge
]
)
+25 -11
View File
@@ -4,9 +4,10 @@ Pytest configuration and shared fixtures for integration tests.
This file configures the Python path to allow importing backend modules
and provides shared fixtures for all tests.
"""
from typing import Generator, Callable, Protocol
from typing import Any, Generator, Callable, Protocol, TypeVar
import pytest
import requests
from tests.test_remote_repo import RemoteRepoHelper
@@ -26,25 +27,38 @@ import pytest
class EventuallyFn(Protocol):
def __call__(
self,
assert_fn: Callable[[], bool],
timeout: float = 2.0,
interval: float = 0.05,
assert_fn: Callable[[], None],
timeout: float = 5.0,
interval: float = 0.5,
) -> None: ...
T = TypeVar('T')
def assert_equal(a: T, b:T, message: str) -> None:
if a != b:
raise AssertionError(f"{message}: {a} != {b}")
def check_state(url: str, predicate: Callable[[Any], None]) -> None:
response = requests.get(f"{url}/state", timeout=5)
assert response.status_code == 200, f"Expected status code 200, got {response.status_code}"
state_data = response.json()
try:
predicate(state_data)
except Exception as e:
raise AssertionError(f"Predicate check failed with error: {e}\nState data: {state_data}") from e
@pytest.fixture
def eventually() -> EventuallyFn:
def _eventually(assert_fn: Callable[[], bool], timeout: float = 2.0, interval: float = 0.05) -> None:
def _eventually(assert_fn: Callable[[], None], timeout: float = 5.0, interval: float = 0.5) -> None:
deadline = time.monotonic() + timeout
last_error = None
last_error: BaseException | None = None
while time.monotonic() < deadline:
try:
res = assert_fn()
if res:
return
except AssertionError as err:
assert_fn()
return
except Exception as err:
last_error = err
time.sleep(interval)
time.sleep(interval)
raise AssertionError(f"Condition not met within {timeout:.2f}s") from last_error
-79
View File
@@ -1,79 +0,0 @@
# import asyncio
import json
import logging
import threading
from typing import Any, Generator
import docker
import pytest
import requests
from app import App
from conftest import EventuallyFn # type: ignore[import-not-found]
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
class TestDockerContainer:
@pytest.fixture(autouse=True)
def setup_teardown(self) -> Generator[None, Any, None]:
"""Set up and tear down test environment"""
# Setup
yield None
# Teardown
client = docker.from_env()
container = client.containers.get("brencher_plain-container")
container.stop(timeout=10)
container.remove(force=True)
logger.info("Container brencher_plain-container stopped and removed")
@pytest.mark.asyncio
async def test_start(self, eventually: EventuallyFn) -> None: # type: ignore[no-any-unimported]
logger.info(f"Starting")
app = App(cli_env_ids_str="brencher_local1")
# app.runWeb(5001)
processing = threading.Thread(target=lambda: app.runWeb(5001), daemon=True)
processing.start()
eventually(
lambda: requests.get("http://localhost:5001/state", timeout=5000).status_code == 200,
20.0,
1.0,
)
# await asyncio.sleep(5000)
state_data = requests.get("http://localhost:5001/state", timeout=5000).json()
logger.info(f"Application state: {json.dumps(state_data, indent=2)}")
branches_data = requests.get("http://localhost:5001/branches", timeout=5000).json()
logger.info(f"Application branches: {json.dumps(branches_data, indent=2)}")
assert "brencher_local1" in state_data, "Missing 'brencher_local1' field in response"
assert len(state_data["brencher_local1"]["pipeline"]) > 0, "Steps defined for 'brencher_local1' are empty"
assert "brencher_local1" in branches_data, "Missing 'brencher_local1' field in response"
assert len(branches_data["brencher_local1"]) > 0, "No branches found for 'brencher_local1'"
# TODO Verify DockerContainerDeploy step status is running and has correct image and ports
# {
# "container_id": "213d161d7d08a0a4a45d666c2f31ad186a212e156ef310673cea24ff9eaa99d7",
# "container_name": "brencher_plain-container",
# "image": "brencher_plain:auto-0c711a54",
# "status": "running",
# "ports": {
# "5001/tcp": [
# {
# "HostIp": "0.0.0.0",
# "HostPort": "5002"
# },
# {
# "HostIp": "::",
# "HostPort": "5002"
# }
# ]
# }
# }
+110
View File
@@ -0,0 +1,110 @@
import logging
from contextlib import contextmanager
from typing import Generator
from playwright.sync_api import Locator, Page, sync_playwright
logger = logging.getLogger(__name__)
class BrencherPage:
def __init__(self, page: Page, env_id: str, action_logger: logging.Logger) -> None:
self._page = page
self._env_id = env_id
self._logger = action_logger
def _dry_run_button(self) -> Locator:
button = self._page.locator(f'h3:has-text("{self._env_id}") button.dry-run-btn')
button.wait_for(state="visible", timeout=10000)
return button
def verify_dry_run_on(self) -> None:
btn_title = self._dry_run_button().get_attribute("title") or ""
assert "Dry run on" in btn_title, f"Expected dry run to be active, got title: {btn_title!r}"
self._logger.info(f"Dry-run button title before click: {btn_title!r}")
def set_dry_run_off(self) -> None:
self._dry_run_button().click()
self._logger.info("Clicked dry-run button to disable dry run")
def click_refresh(self) -> None:
self._page.locator("#refresh-branches").click()
self._logger.info("Clicked refresh button")
def verify_dry_run_off(self) -> None:
self._page.wait_for_function(
"""(targetEnvId) => {
const h3s = [...document.querySelectorAll('h3.env-title')];
const h3 = h3s.find(el => el.textContent.includes(targetEnvId));
if (!h3) return false;
const btn = h3.querySelector('button.dry-run-btn');
return btn && !btn.title.includes('Dry run on');
}""",
arg=self._env_id,
timeout=10000,
)
def verify_pipeline_step_status(self, step_name: str, expected_status: str, timeout_ms: int = 10000) -> None:
"""
Verify that a pipeline step reaches the expected status by polling the page DOM.
Args:
step_name: Name of the pipeline step (e.g., "DockerContainerDeploy")
expected_status: Expected status value ("running", "ok", "error", or "dry-run")
timeout_ms: Timeout in milliseconds (default 10000)
"""
# JavaScript to check pipeline step status
js_check = """(stepName, expectedStatus) => {
const jobItems = document.querySelectorAll('.job-item');
for (const item of jobItems) {
const headerText = item.querySelector('.job-header')?.textContent || '';
if (headerText.includes(stepName)) {
const spinner = item.querySelector('.step-spinner');
const errorIndicator = item.querySelector('span[title="Error"]');
if (expectedStatus === 'running') {
return spinner !== null;
} else if (expectedStatus === 'ok') {
return spinner === null && errorIndicator === null;
} else if (expectedStatus === 'error') {
return errorIndicator !== null;
} else if (expectedStatus === 'dry-run') {
// For dry-run, check the spoiler content
const spoiler = item.querySelector('.job-spoiler');
if (spoiler && spoiler.style.display !== 'none') {
const statusText = spoiler.textContent;
return statusText.includes('dry-run') || statusText.includes('dry_run');
}
return false;
}
}
}
return false;
}"""
self._page.wait_for_function(
js_check,
arg=[step_name, expected_status],
timeout=timeout_ms,
)
self._logger.info(f"Pipeline step '{step_name}' reached status '{expected_status}'")
@contextmanager
def brencher_page(
app_link: str,
env_id: str,
action_logger: logging.Logger | None = None,
) -> Generator[BrencherPage, None, None]:
active_logger = action_logger or logger
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=True)
page = browser.new_page()
page.goto(app_link)
page.wait_for_load_state("networkidle")
try:
yield BrencherPage(page, env_id, active_logger)
finally:
browser.close()
+52
View File
@@ -0,0 +1,52 @@
import asyncio
import sys
import types
from typing import Callable
import pytest
from enironment import Environment
import processing
from app import App
class TestAppAsyncio:
@pytest.mark.asyncio
async def test_run_web_async_stops_processing_when_server_exits(self, monkeypatch: pytest.MonkeyPatch) -> None:
process_calls = 0
emit_calls = 0
processing_started = asyncio.Event()
loop = asyncio.get_running_loop()
web_app_args: dict[str, object] = {}
def fake_process_all_jobs(environments: list[Environment], onupdate: Callable[[], None]) -> bool:
nonlocal process_calls
process_calls += 1
assert environments == []
onupdate()
loop.call_soon_threadsafe(processing_started.set)
return False
class FakeWebApp:
def __init__(self, core: App, port: int) -> None:
web_app_args["core"] = core
web_app_args["port"] = port
def emit_envs(self) -> None:
nonlocal emit_calls
emit_calls += 1
async def start_async(self) -> None:
await asyncio.wait_for(processing_started.wait(), timeout=1)
monkeypatch.setattr(processing, "process_all_jobs", fake_process_all_jobs)
monkeypatch.setitem(sys.modules, "web", types.SimpleNamespace(WebApp=FakeWebApp))
app = App({})
await asyncio.wait_for(app.runWebAsync(5001), timeout=1)
assert process_calls >= 1
assert emit_calls >= 1
assert web_app_args == {"core": app, "port": 5001}
assert app.emit_callback is None
+16 -3
View File
@@ -1,5 +1,5 @@
import pytest
from enironment import wrap_in_cached
from enironment import wrap_in_cached, SharedStateConflictError
from tests.test_remote_repo import RemoteRepoHelper
@@ -15,13 +15,26 @@ class TestCachedEnv:
repo_helper.env = wrap_in_cached(repo_helper.env)
repo_helper.env.branches = []
repo_helper.set_desired_branches([])
with pytest.raises(BaseException, match="Empty branches set"):
repo_helper.checkout_merged.progress()
repo_helper.env.branches = [('master', 'HEAD')]
repo_helper.set_desired_branches([('master', 'HEAD')])
result = repo_helper.checkout_merged.progress()
# Verify same auto branch is used with actual value checks
assert result.commit_hash == commit1.hexsha, f"Expected {commit1.hexsha}, got {result.commit_hash}"
def test_inmemory_state_rejects_stale_token(self, repo_helper: RemoteRepoHelper) -> None:
_ = repo_helper.create_commit(repo_helper.repo, "master", "master", "file1.txt", "content1", "Initial commit")
_ = repo_helper.create_commit(repo_helper.repo, "master", "branch1", "file2.txt", "content2", "Branch1 commit")
initial_state = repo_helper.resolve_initial_branches.progress()
initial_token = initial_state.token
assert isinstance(initial_token, str)
repo_helper.resolve_initial_branches.set_branches([("master", "HEAD")], expected_token=initial_token)
with pytest.raises(SharedStateConflictError, match="State token mismatch"):
repo_helper.resolve_initial_branches.set_branches([("branch1", "HEAD")], expected_token=initial_token)
+168
View File
@@ -0,0 +1,168 @@
"""
Unit tests for CachingStep input-hash-based cache invalidation.
"""
from typing import Any
import pytest
from enironment import AbstractStep, Environment
from steps.shared_state import SharedStateHolderInMemory
from steps.step import CachingStep, NotReadyException
class ConstantStep(AbstractStep[str]):
"""Step that always returns a fixed value, counting calls."""
def __init__(self, value: str) -> None:
super().__init__()
self.value = value
self.call_count = 0
def progress(self) -> str:
self.call_count += 1
return self.value
class DependentStep(AbstractStep[str]):
"""Step that depends on another AbstractStep and appends a suffix."""
def __init__(self, upstream: AbstractStep[str], suffix: str = "-processed") -> None:
super().__init__()
self.upstream = upstream
self.suffix = suffix
self.call_count = 0
def progress(self) -> str:
self.call_count += 1
return self.upstream.progress() + self.suffix
class FailingStep(AbstractStep[str]):
"""Step that always raises an exception."""
def __init__(self) -> None:
super().__init__()
self.call_count = 0
def progress(self) -> str:
self.call_count += 1
raise RuntimeError("step failed")
def _make_env(steps: list[AbstractStep[Any]]) -> Environment:
return Environment(id="test", state=SharedStateHolderInMemory(unmerge=None), pipeline=steps)
class TestCachingStep:
def test_executes_on_first_call(self) -> None:
inner = ConstantStep("hello")
cached = CachingStep(inner)
_make_env([cached])
result = cached.progress()
assert result == "hello"
assert inner.call_count == 1
def test_caches_result_when_inputs_unchanged(self) -> None:
inner = ConstantStep("hello")
cached = CachingStep(inner)
_make_env([cached])
cached.progress()
cached.progress()
cached.progress()
assert inner.call_count == 1
def test_reexecutes_when_upstream_output_changes(self) -> None:
upstream = ConstantStep("v1")
inner = DependentStep(upstream)
cached_upstream = CachingStep(upstream)
cached = CachingStep(inner)
_make_env([cached_upstream, cached])
# Wire up: replace inner's `upstream` reference with the cached wrapper
inner.upstream = cached_upstream
result1 = cached.progress()
assert result1 == "v1-processed"
assert inner.call_count == 1
# Change the upstream value so the hash changes
upstream.value = "v2"
# Force the upstream cache to re-execute so it returns the new value
cached_upstream.reset()
result2 = cached.progress()
assert result2 == "v2-processed"
assert inner.call_count == 2
def test_does_not_reexecute_when_upstream_output_unchanged(self) -> None:
upstream = ConstantStep("stable")
inner = DependentStep(upstream)
cached_upstream = CachingStep(upstream)
cached = CachingStep(inner)
_make_env([cached_upstream, cached])
inner.upstream = cached_upstream
cached.progress()
cached.progress()
cached.progress()
assert inner.call_count == 1
def test_reset_forces_reexecution(self) -> None:
inner = ConstantStep("hello")
cached = CachingStep(inner)
_make_env([cached])
cached.progress()
assert inner.call_count == 1
cached.reset()
cached.progress()
assert inner.call_count == 2
def test_retries_after_exception(self) -> None:
inner = FailingStep()
cached = CachingStep(inner)
_make_env([cached])
with pytest.raises(RuntimeError):
cached.progress()
assert inner.call_count == 1
# Second call should retry because the result is still an exception
with pytest.raises(RuntimeError):
cached.progress()
assert inner.call_count == 2
def test_initial_state_raises_not_ready(self) -> None:
inner = ConstantStep("hello")
cached = CachingStep(inner)
# Do NOT set up env – but CachingStep itself should start with NotReadyException
assert isinstance(cached._result, NotReadyException)
def test_leaf_step_executes_once(self) -> None:
"""A step with no AbstractStep dependencies uses a constant hash; executes only once."""
inner = ConstantStep("leaf")
cached = CachingStep(inner)
_make_env([cached])
cached.progress()
cached.progress()
assert inner.call_count == 1
def test_reset_clears_hash(self) -> None:
inner = ConstantStep("x")
cached = CachingStep(inner)
_make_env([cached])
cached.progress()
assert cached._input_hash is not None
cached.reset()
assert cached._input_hash is None
+82
View File
@@ -0,0 +1,82 @@
import asyncio
import json
import logging
from contextlib import suppress
from typing import Any, Generator
import docker
import pytest
import requests
from app import App
from configs import brencher_local1
from .conftest import EventuallyFn, assert_equal
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
class TestDockerContainer:
@pytest.fixture(autouse=True)
def setup_teardown(self) -> Generator[None, Any, None]:
"""Set up and tear down test environment"""
# Setup
yield None
# Teardown
client = docker.from_env()
container = client.containers.get("brencher_plain-container")
container.stop(timeout=10)
container.remove(force=True)
logger.info("Container brencher_plain-container stopped and removed")
@pytest.mark.skip
@pytest.mark.asyncio
async def test_start(self, eventually: EventuallyFn) -> None:
logger.info(f"Starting")
app = App({"brencher_local1": brencher_local1.brencher_local1})
processing = asyncio.create_task(app.runWebAsync(5001))
try:
eventually(
lambda: assert_equal(requests.get("http://localhost:5001/state", timeout=5000).status_code, 200, "Server did not respond with 200 OK within timeout"),
20.0,
1.0,
)
state_data = requests.get("http://localhost:5001/state", timeout=5000).json()
logger.info(f"Application state: {json.dumps(state_data, indent=2)}")
branches_data = requests.get("http://localhost:5001/branches", timeout=5000).json()
logger.info(f"Application branches: {json.dumps(branches_data, indent=2)}")
assert "brencher_local1" in state_data, "Missing 'brencher_local1' field in response"
assert len(state_data["brencher_local1"]["pipeline"]) > 0, "Steps defined for 'brencher_local1' are empty"
assert "brencher_local1" in branches_data, "Missing 'brencher_local1' field in response"
assert len(branches_data["brencher_local1"]) > 0, "No branches found for 'brencher_local1'"
finally:
processing.cancel()
with suppress(asyncio.CancelledError):
await processing
assert processing.cancelled()
# TODO Verify DockerContainerDeploy step status is running and has correct image and ports
# {
# "container_id": "213d161d7d08a0a4a45d666c2f31ad186a212e156ef310673cea24ff9eaa99d7",
# "container_name": "brencher_plain-container",
# "image": "brencher_plain:auto-0c711a54",
# "status": "running",
# "ports": {
# "5001/tcp": [
# {
# "HostIp": "0.0.0.0",
# "HostPort": "5002"
# },
# {
# "HostIp": "::",
# "HostPort": "5002"
# }
# ]
# }
# }
+104
View File
@@ -0,0 +1,104 @@
import logging
import threading
from typing import Any, Callable, Generator, TypeVar
import docker
import pytest
import requests
from app import App
from .conftest import EventuallyFn, assert_equal, check_state
from .playwright_helper import brencher_page
from tests.configs import nginx
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
APP_PORT = 5003
APP_URL = f"http://localhost:{APP_PORT}"
CONTAINER_NAME = "test_plain-container"
class TestDryRunPlaywright:
@pytest.fixture(autouse=True)
def setup_teardown(self) -> Generator[None, Any, None]:
def stop_and_remove_container() -> None:
client = docker.from_env()
try:
container = client.containers.get(CONTAINER_NAME)
container.stop(timeout=10)
container.remove(force=True)
logger.info(f"Container {CONTAINER_NAME} stopped and removed")
except Exception:
logger.info(f"Container {CONTAINER_NAME} not found, no need to stop/remove")
stop_and_remove_container()
yield None
stop_and_remove_container()
def test_dry_run_prevents_deploy_until_disabled(self, eventually: EventuallyFn) -> None:
# Start app with nginx_local1 with main branch pre-selected and dry mode on.
env = nginx.test_local1
env.state.set_branches([(nginx.CURRENT_BRANCH, nginx.CURRENT_COMMIT)])
env.state.set_dry(True)
app = App({env.id: env})
server_thread = threading.Thread(target=lambda: app.runWeb(APP_PORT), daemon=True)
server_thread.start()
try:
# Wait for server to be up
eventually(lambda: assert_equal(requests.get(f"{APP_URL}/state", timeout=5).status_code, 200, "Server did not respond with 200 OK within timeout"))
logger.info("Server is up")
# Wait for the pipeline to stabilize with dry=True:
# DockerContainerDeploy should return status="dry-run"
with brencher_page(APP_URL, env.id) as page:
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] == "SharedStateHolderInMemory"
)["status"]['dry'], True, "Pipeline step status is not 'running'")),
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "dry-run")
logger.info("Pipeline is in dry-run mode")
# Verify no real container was created
docker_client = docker.from_env()
real_containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(real_containers) == 0, "Container should NOT exist during dry run"
# Disable dry run from the UI.
page.verify_dry_run_on()
page.set_dry_run_off()
page.verify_dry_run_off()
# Wait for pipeline to re-run and actually deploy the container
eventually(
lambda: check_state(APP_URL, lambda s: assert_equal(next(
p for p in s[env.id]["pipeline"] if p["name"] =="SharedStateHolderInMemory"
)["status"]['dry'], False, "Pipeline step status is not 'running'")),
)
# Wait for the real container to be created and started.
eventually(
lambda: _assert_container_running(docker_client),
# Allow time for a cold image build and first container start on CI runners.
timeout=180,
interval=5.0,
)
# page.verify_pipeline_step_status("DockerContainerDeploy", "running")
logger.info("Container deployed and running after dry run disabled")
# Final sanity: real container must exist and be running
containers = docker_client.containers.list(filters={"name": CONTAINER_NAME})
assert len(containers) == 1, "Container should exist and be running after deploy"
assert containers[0].status == "running"
finally:
app.stop()
server_thread.join(timeout=30)
if server_thread.is_alive():
logger.warning("App server thread did not stop within 30 seconds")
def _assert_container_running(docker_client: docker.DockerClient) -> None:
containers = docker_client.containers.list(filters={"name": CONTAINER_NAME}, all=True)
assert len(containers) > 0, "Container should exist after deploy"
assert_equal({container.status for container in containers}, {"running"}, "Container should be running after deploy")
+34 -28
View File
@@ -8,7 +8,7 @@ import os
import git
import pytest
from steps.git import CheckoutAndMergeResult
from steps.git import CheckoutAndMergeResult, GitUnmergeResult
from .test_remote_repo import RemoteRepoHelper
@@ -25,7 +25,7 @@ class TestGitIntegration:
commit3 = repo_helper.create_commit(repo_helper.repo, "master", "branch2", "file3.txt", "content3",
"Branch2 commit") # noqa: F841
repo_helper.env.branches = [("branch1", "HEAD")]
repo_helper.set_desired_branches([("branch1", "HEAD")])
result = repo_helper.checkout_merged.progress()
assert isinstance(result, CheckoutAndMergeResult)
@@ -37,9 +37,9 @@ class TestGitIntegration:
('file1.txt', 'content1'),
('file2.txt', 'content2'),
])
assert repo_helper.git_unmerge.progress() == [('branch1', commit2.hexsha)], f"Invalid Unmerge result"
assert repo_helper.git_unmerge.progress().branches == [('branch1', commit2.hexsha)], f"Invalid Unmerge result"
repo_helper.env.branches = [("branch2", "HEAD")]
repo_helper.set_desired_branches([("branch2", "HEAD")])
result = repo_helper.checkout_merged.progress()
assert isinstance(result, CheckoutAndMergeResult)
@@ -52,7 +52,7 @@ class TestGitIntegration:
('file3.txt', 'content3'),
])
print(commit1.hexsha, commit2.hexsha, commit3.hexsha)
assert repo_helper.git_unmerge.progress() == [("branch2", commit3.hexsha)], f"Invalid Unmerge result"
assert repo_helper.git_unmerge.progress().branches == [("branch2", commit3.hexsha)], f"Invalid Unmerge result"
def test_checkout_merged_two_branches(self, repo_helper: RemoteRepoHelper) -> None:
"""Test merging two branches successfully"""
@@ -66,7 +66,7 @@ class TestGitIntegration:
"Branch2 commit") # noqa: F841
# Create a GitClone-like working directory
repo_helper.env.branches = [("branch1", "HEAD"), ("branch2", "HEAD")]
repo_helper.set_desired_branches([("branch1", "HEAD"), ("branch2", "HEAD")])
result = repo_helper.checkout_merged.progress()
# Verify result with specific field checks
@@ -99,7 +99,7 @@ class TestGitIntegration:
commit4 = repo_helper.create_commit(repo_helper.repo, "master", "branch3", "file4.txt", "content4",
"Branch3 commit") # noqa: F841
repo_helper.env.branches = [("branch1", "HEAD"), ("branch2", "HEAD"), ("branch3", "HEAD")]
repo_helper.set_desired_branches([("branch1", "HEAD"), ("branch2", "HEAD"), ("branch3", "HEAD")])
result = repo_helper.checkout_merged.progress()
@@ -139,7 +139,7 @@ class TestGitIntegration:
commit3 = repo_helper.create_commit(repo_helper.repo, "branch1", "branch1", "file3.txt", "content3",
"Branch1 commit 2") # noqa: F841
repo_helper.env.branches = [("master", "HEAD"), ("branch1", "HEAD")]
repo_helper.set_desired_branches([("master", "HEAD"), ("branch1", "HEAD")])
result = repo_helper.checkout_merged.progress()
@@ -165,7 +165,7 @@ class TestGitIntegration:
"Branch2 change") # noqa: F841
# Create environment
repo_helper.env.branches = [("branch1", "HEAD"), ("branch2", "HEAD")]
repo_helper.set_desired_branches([("branch1", "HEAD"), ("branch2", "HEAD")])
# Expect merge conflict to be raised as BaseException
with pytest.raises(BaseException, match="Merge conflict"):
@@ -184,7 +184,7 @@ class TestGitIntegration:
commit3 = repo_helper.create_commit(repo_helper.repo, "master", "branch2", "file3.txt", "content3",
"Branch2 commit") # noqa: F841
repo_helper.env.branches = [("branch1", "HEAD"), ("branch2", "HEAD")]
repo_helper.set_desired_branches([("branch1", "HEAD"), ("branch2", "HEAD")])
result1 = repo_helper.checkout_merged.progress()
auto_branch_name = result1.remote_branch_name
@@ -212,7 +212,7 @@ class TestGitIntegration:
commit3 = repo_helper.create_commit(repo_helper.repo, "master", "branch2", "file3.txt", "content3",
"Branch2 commit")
repo_helper.env.branches = []
repo_helper.set_desired_branches([])
# Mock DockerSwarmCheck with version string
version_str = f"auto-{commit2.hexsha[:8]}-{commit3.hexsha[:8]}"
@@ -221,24 +221,30 @@ class TestGitIntegration:
# Test GitUnmerge
result = repo_helper.git_unmerge.progress()
# Verify result contains branch information with specific value checks
assert isinstance(result, list), "Result should be a list"
assert len(result) > 0, "Result should not be empty"
# Verify result is a GitUnmergeResult with branch information
assert isinstance(result, GitUnmergeResult), "Result should be a GitUnmergeResult"
assert len(result.branches) > 0, "branches should not be empty"
for branch_name, commit_hash in result:
for branch_name, commit_hash in result.branches:
assert isinstance(branch_name, str), f"Branch name should be string, got {type(branch_name)}"
assert len(branch_name) > 0, "Branch name should not be empty"
assert isinstance(commit_hash, str), f"Commit hash should be string, got {type(commit_hash)}"
assert len(commit_hash) == 40, f"Commit hash should be 40 chars, got {len(commit_hash)}: {commit_hash}"
assert commit_hash.isalnum(), f"Commit hash should be alphanumeric: {commit_hash}"
assert "Deployed" in result.columns, "GitUnmergeResult should have a 'Deployed' column"
for branch_name, short_hash in result.columns["Deployed"].items():
assert isinstance(branch_name, str)
assert isinstance(short_hash, str)
assert len(short_hash) == 8, f"Deployed column should contain 8-char short hashes, got: {short_hash}"
def test_git_unmerge_invalid_version(self, repo_helper: RemoteRepoHelper) -> None:
"""Test GitUnmerge with invalid version format"""
commit1 = repo_helper.create_commit(repo_helper.repo, "master", "master", "file1.txt", "content1",
"Initial commit") # noqa: F841
repo_helper.env.branches = []
repo_helper.set_desired_branches([])
version_str = "invalid-version-format"
repo_helper.mock_check.version = lambda: version_str # type: ignore[attr-defined]
@@ -256,22 +262,22 @@ class TestGitIntegration:
commit3 = repo_helper.create_commit(repo_helper.repo, "branch1", "branch1", "file3.txt", "content3",
"Branch1 commit 2") # noqa: F841
repo_helper.env.branches = []
repo_helper.set_desired_branches([])
repo_helper.mock_check.version = lambda: f"auto-{commit2.hexsha[:8]}" # type: ignore[attr-defined]
result = repo_helper.git_unmerge.progress()
assert isinstance(result, list)
assert len(result) >= 1
assert isinstance(result, GitUnmergeResult)
assert len(result.branches) >= 1
# Extract branch names and commit hashes
result_dict = {commit_hash: branch_name for branch_name, commit_hash in result}
result_dict = {commit_hash: branch_name for branch_name, commit_hash in result.branches}
assert commit2.hexsha in result_dict, \
f"Expected commit2 {commit2.hexsha} (non-HEAD commit in branch1) to be in results. " \
f"GitUnmerge should return branch1 even when commit2 is not HEAD. " \
f"Current results: {result}"
f"Current results: {result.branches}"
assert result_dict[commit2.hexsha] == "branch1", \
f"branch1 should be associated with commit2. Got: {result_dict.get(commit2.hexsha)}"
@@ -291,18 +297,18 @@ class TestGitIntegration:
commit5 = repo_helper.create_commit(repo_helper.repo, "branch2", "branch2", "file5.txt", "content5",
"Branch2 commit 2")
repo_helper.env.branches = []
repo_helper.set_desired_branches([])
repo_helper.mock_check.version = lambda: f"auto-{commit2.hexsha[:8]}-{commit4.hexsha[:8]}" # type: ignore[attr-defined]
result = repo_helper.git_unmerge.progress()
# Verify result contains branch information
assert isinstance(result, list)
assert len(result) >= 1 # At least branch2 should be found
assert isinstance(result, GitUnmergeResult)
assert len(result.branches) >= 1 # At least branch2 should be found
# Extract branch names and commit hashes
result_dict = {commit_hash: branch_name for branch_name, commit_hash in result}
result_dict = {commit_hash: branch_name for branch_name, commit_hash in result.branches}
# commit4 should be in the results (HEAD of branch2)
assert commit4.hexsha in result_dict, f"Expected commit4 {commit4.hexsha} (HEAD of branch2) in results"
@@ -316,7 +322,7 @@ class TestGitIntegration:
assert commit2.hexsha in result_dict, \
f"Expected commit2 {commit2.hexsha} (non-HEAD commit in branch1) to be in results. " \
f"GitUnmerge should return branch1 even when commit2 is not HEAD. " \
f"Current results: {result}"
f"Current results: {result.branches}"
assert result_dict[commit2.hexsha] == "branch1", \
f"branch1 should be associated with commit2. Got: {result_dict.get(commit2.hexsha)}"
@@ -327,12 +333,12 @@ class TestGitIntegration:
commit1 = repo_helper.create_commit(repo_helper.repo, "master", "master", "file1.txt", "content1",
"Initial commit") # noqa: F841
repo_helper.env.branches = []
repo_helper.set_desired_branches([])
with pytest.raises(BaseException, match="Empty branches set"):
repo_helper.checkout_merged.progress()
repo_helper.env.branches = [('master', 'HEAD')]
repo_helper.set_desired_branches([('master', 'HEAD')])
result = repo_helper.checkout_merged.progress()
# Verify same auto branch is used with actual value checks
+42
View File
@@ -0,0 +1,42 @@
import json
import pytest
from enironment import SharedStateConflictError
from processing import process_all_jobs
from steps.git import GitClone
from steps.shared_state import SharedStateHolderInGit
from tests.test_remote_repo import RemoteRepoHelper
class TestInitialBranches:
"""Integration tests for git operations"""
def test_resolve_initial_branches(self, repo_helper: RemoteRepoHelper) -> None:
_ = repo_helper.create_commit(repo_helper.repo, "master", "master", "file1.txt", "content1", "Initial commit")
_ = repo_helper.create_commit(repo_helper.repo, "master", "branch1", "file2.txt", "content2", "Branch1 commit")
_ = repo_helper.create_commit(repo_helper.repo, "state", "state", "blank.txt", "", "Branch1 commit")
clone = GitClone(url=repo_helper.remote_dir)
clone.env = repo_helper.env
cloneState = GitClone(url=repo_helper.remote_dir, n='state')
cloneState.env = repo_helper.env
resolve_step = SharedStateHolderInGit(clone, state_repo=cloneState, state_branch="state")
resolve_step.env = repo_helper.env
stale_token = resolve_step.progress().token
assert isinstance(stale_token, str)
t = resolve_step.set_branches([("branch1", "HEAD")], expected_token=stale_token).token
t = resolve_step.set_dry(True, expected_token=t).token
with pytest.raises(SharedStateConflictError, match="State token mismatch"):
resolve_step.set_branches([("master", "HEAD")], expected_token=stale_token)
t = resolve_step.set_branches([("master", "HEAD")], expected_token=t).token
t = resolve_step.set_dry(False, expected_token=t).token
jpl = repo_helper.repo.git.show(f"state:{repo_helper.env.id}.json")
pl = json.loads(jpl)
assert pl['branches'] == [["master", "HEAD"]]
assert pl['dry'] == False
+149 -126
View File
@@ -7,157 +7,180 @@ git operations in an isolated environment.
import os
import shutil
import tempfile
from typing import Dict, Tuple, List, Callable
from typing import Dict, Tuple, List, Callable, cast
import git
from enironment import Environment, AbstractStep
from enironment import Environment, AbstractStep, SharedStateHolder, get_step
from steps.docker import DockerSwarmCheckResult
from steps.git import CheckoutAndMergeResult, CheckoutMerged, GitUnmerge, GitClone, HasVersion
from steps.git import CheckoutAndMergeResult, CheckoutMerged, GitUnmerge, GitClone, HasVersion, GitUnmergeResult
from steps.shared_state import SharedStateHolderInMemory
from steps.step import CachingStep # noqa: F401
class MockDockerSwarmCheck(AbstractStep[Dict[str, HasVersion]]):
"""Mock DockerSwarmCheck object for testing"""
"""Mock DockerSwarmCheck object for testing"""
def __init__(self, version: Callable[[], str]) -> None:
super().__init__()
self.version = version
self.stack_name = "test_stack"
def __init__(self, version: Callable[[], str]) -> None:
super().__init__()
self.version = version
self.stack_name = "test_stack"
def progress(self) -> Dict[str, HasVersion]:
return {
"service1": DockerSwarmCheckResult(
name="service1",
version=self.version(),
image="test:latest",
stack="test-stack"
)
}
def progress(self) -> Dict[str, HasVersion]:
return {
"service1": DockerSwarmCheckResult(
name="service1",
version=self.version(),
image="test:latest",
stack="test-stack"
)
}
class RemoteRepoHelper:
"""Helper class for creating and managing test git repositories"""
env: Environment
"""Helper class for creating and managing test git repositories"""
env: Environment
def __init__(self) -> None:
self.remote_dir = tempfile.mkdtemp(prefix="test_remote_")
self.local_dir = tempfile.mkdtemp(prefix="test_local_")
self.repo = git.Repo.init(self.remote_dir, bare=False)
# Configure git user
with self.repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
git_clone = GitClone(repo_path=self.local_dir)
checkout_merged = CheckoutMerged(
wd=git_clone,
git_user_email="test@example.com",
git_user_name="Test User",
push=False,
)
mock_check = MockDockerSwarmCheck(lambda: f"auto-{self.checkout_merged.progress().version}")
git_unmerge = GitUnmerge(
wd=git_clone,
check=mock_check,
)
self.env = Environment(
id="test1",
branches=[],
dry=False,
repo=self.remote_dir,
pipeline=[
git_clone,
checkout_merged,
mock_check,
git_unmerge
]
)
def __init__(self) -> None:
self.remote_dir = tempfile.mkdtemp(prefix="test_remote_")
self.local_dir = tempfile.mkdtemp(prefix="test_local_")
self.repo = git.Repo.init(self.remote_dir, bare=False)
# Configure git user
with self.repo.config_writer() as cw:
cw.set_value("user", "email", "test@example.com")
cw.set_value("user", "name", "Test User")
git_clone = GitClone(url=self.remote_dir, repo_path=self.local_dir)
mock_check = MockDockerSwarmCheck(lambda: f"auto-{self.checkout_merged.progress().version}")
git_unmerge = GitUnmerge(
wd=git_clone,
check=mock_check,
)
state = SharedStateHolderInMemory(unmerge=None)
checkout_merged = CheckoutMerged(
wd=git_clone,
desired_branches=state,
git_user_email="test@example.com",
git_user_name="Test User",
push=False,
)
self.env = Environment(
id="test1",
state=state,
pipeline=[
git_clone,
state,
checkout_merged,
mock_check,
git_unmerge
]
)
@property
def checkout_merged(self) -> AbstractStep[CheckoutAndMergeResult]:
return [i for i in self.env.pipeline if
isinstance(i, CheckoutMerged) or (isinstance(i, CachingStep) and isinstance(i.step, CheckoutMerged))][0]
@property
def checkout_merged(self) -> AbstractStep[CheckoutAndMergeResult]:
return get_step(self.env.pipeline, CheckoutMerged)
@property
def git_clone(self) -> AbstractStep[str]:
return [i for i in self.env.pipeline if
isinstance(i, GitClone) or (isinstance(i, CachingStep) and isinstance(i.step, GitClone))][0]
@property
def resolve_initial_branches(self) -> SharedStateHolder:
return self.env.state
@property
def git_unmerge(self) -> AbstractStep[List[Tuple[str, str]]]:
return [i for i in self.env.pipeline if
isinstance(i, GitUnmerge) or (isinstance(i, CachingStep) and isinstance(i.step, GitUnmerge))][0]
def set_desired_branches(self, branches: List[Tuple[str, str]]) -> None:
step: SharedStateHolder = get_step(self.env.pipeline, SharedStateHolder) # type: ignore[type-abstract]
if isinstance(step, CachingStep) and isinstance(step._step, SharedStateHolder):
step._step.set_branches(branches)
step.reset()
else:
step.set_branches(branches)
@property
def mock_check(self) -> AbstractStep[Dict[str, HasVersion]]:
return [i for i in self.env.pipeline if isinstance(i, MockDockerSwarmCheck) or (
isinstance(i, CachingStep) and isinstance(i.step, MockDockerSwarmCheck))][0]
def get_desired_branches(self) -> List[Tuple[str, str]]:
return self.resolve_initial_branches.progress().branches
def teardown(self) -> None:
"""Clean up temporary directories"""
if self.remote_dir:
shutil.rmtree(self.remote_dir, ignore_errors=True)
if self.local_dir:
shutil.rmtree(self.local_dir, ignore_errors=True)
@property
def git_clone(self) -> AbstractStep[str]:
return get_step(self.env.pipeline, GitClone)
def create_commit(self, repo: git.Repo, from_branch: str, to_branch: str, filename: str, content: str,
message: str) -> git.Commit:
"""Create a commit in the repository
@property
def git_unmerge(self) -> AbstractStep[GitUnmergeResult]:
return get_step(self.env.pipeline, GitUnmerge)
If to_branch doesn't exist, it will be created from from_branch and checked out.
"""
if from_branch in [head.name for head in repo.heads]:
repo.heads[from_branch].checkout()
if from_branch != to_branch:
if to_branch not in [head.name for head in repo.heads]:
new_branch = repo.create_head(to_branch)
new_branch.checkout()
else:
raise BaseException("Target branch already exists.")
@property
def mock_check(self) -> AbstractStep[Dict[str, HasVersion]]:
return get_step(self.env.pipeline, MockDockerSwarmCheck)
# Create commit
file_path = os.path.join(repo.working_dir, filename)
with open(file_path, 'w') as f:
f.write(content)
repo.index.add([filename])
return repo.index.commit(message)
def teardown(self) -> None:
"""Clean up temporary directories"""
if self.remote_dir:
shutil.rmtree(self.remote_dir, ignore_errors=True)
if self.local_dir:
shutil.rmtree(self.local_dir, ignore_errors=True)
def verify_working_directory_files(self, expected_files: List[Tuple[str, str]]) -> None:
"""Verify that working directory contains exactly the expected files with correct content.
def create_commit(self, repo: git.Repo, from_branch: str, to_branch: str, filename: str, content: str,
message: str) -> git.Commit:
"""Create a commit in the repository
Args:
directory: Path to the working directory to verify
expected_files: List of tuples (filename, expected_content)
"""
files_in_wd = set(os.listdir(self.local_dir)) - {'.git'}
expected_filenames = {filename for filename, _ in expected_files}
assert files_in_wd == expected_filenames, f"Expected files {expected_filenames}, got {files_in_wd}"
If to_branch doesn't exist, it will be created from from_branch and checked out.
"""
if from_branch not in [head.name for head in repo.heads]:
# Create initial commit without parents
file_path = os.path.join(repo.working_dir, filename)
with open(file_path, 'w') as f:
f.write(content)
repo.index.add([filename])
repo.index.commit(message)
repo.create_head(from_branch)
return repo.head.commit
else:
repo.heads[from_branch].checkout()
if from_branch != to_branch:
if to_branch not in [head.name for head in repo.heads]:
new_branch = repo.create_head(to_branch)
new_branch.checkout()
else:
raise BaseException("Target branch already exists.")
for filename, expected_content in expected_files:
with open(os.path.join(self.local_dir, filename)) as f:
actual_content = f.read()
assert actual_content == expected_content, f"{filename} has incorrect content. Expected '{expected_content}', got '{actual_content}'"
# Create commit
file_path = os.path.join(repo.working_dir, filename)
with open(file_path, 'w') as f:
f.write(content)
repo.index.add([filename])
return repo.index.commit(message)
def _safe_git_log(self, repo_path: str, title: str) -> str:
def _as_text(value: object) -> str:
if isinstance(value, bytes):
return value.decode(errors='replace')
return str(value)
def verify_working_directory_files(self, expected_files: List[Tuple[str, str]]) -> None:
"""Verify that working directory contains exactly the expected files with correct content.
repo = git.Repo(repo_path)
commits = list(repo.iter_commits("--all", max_count=30))
refs = [
f"{_as_text(it.commit)[:10]}:{_as_text(it.name)}, is_remote:{it.is_remote()}"
for it in repo.refs
]
lines = [
f"{_as_text(commit.hexsha)[:10]} (p{[_as_text(it.hexsha)[:10] for it in commit.parents]}): {_as_text(commit.message).splitlines()[0]}"
for commit in commits
]
return f"[{title}] git log ({repo_path})\n{"\n".join(lines)}\nrefs:\n{"\n".join(refs)}"
Args:
directory: Path to the working directory to verify
expected_files: List of tuples (filename, expected_content)
"""
files_in_wd = set(os.listdir(self.local_dir)) - {'.git'}
expected_filenames = {filename for filename, _ in expected_files}
assert files_in_wd == expected_filenames, f"Expected files {expected_filenames}, got {files_in_wd}"
def print_git_logs(self) -> None:
"""Print remote and local repository logs for debugging failed tests."""
print(f"Version: {git.__version__}")
for filename, expected_content in expected_files:
with open(os.path.join(self.local_dir, filename)) as f:
actual_content = f.read()
assert actual_content == expected_content, f"{filename} has incorrect content. Expected '{expected_content}', got '{actual_content}'"
print(self._safe_git_log(self.remote_dir, "remote"))
print(self._safe_git_log(self.local_dir, "local"))
def _safe_git_log(self, repo_path: str, title: str) -> str:
def _as_text(value: object) -> str:
if isinstance(value, bytes):
return value.decode(errors='replace')
return str(value)
repo = git.Repo(repo_path)
commits = list(repo.iter_commits("--all", max_count=30))
refs = [
f"{_as_text(it.commit)[:10]}:{_as_text(it.name)}, is_remote:{it.is_remote()}"
for it in repo.refs
]
lines = [
f"{_as_text(commit.hexsha)[:10]} (p{[_as_text(it.hexsha)[:10] for it in commit.parents]}): {_as_text(commit.message).splitlines()[0]}"
for commit in commits
]
return f"[{title}] git log ({repo_path})\n{"\n".join(lines)}\nrefs:\n{"\n".join(refs)}"
def print_git_logs(self) -> None:
"""Print remote and local repository logs for debugging failed tests."""
print(f"Version: {git.__version__}")
print(self._safe_git_log(self.remote_dir, "remote"))
print(self._safe_git_log(self.local_dir, "local"))