feat(connectivity): verify Hermes relocation prerequisites #7
Closed
hermes
wants to merge 10 commits from
feat/hermes-relocation-connectivity into master
pull from: feat/hermes-relocation-connectivity
merge into: :master
:master
:feat/gitea-second-runner-rudolf-t
:feat/hermes_squid-kubernetes-api-proxy
:fix/runner-rudolf-t-internal-gitea
:chore/move-connectivity-tester
:auto-3041db43-d17a9362-e9de263e
:feat/hermes-rudolf-t
:auto-d17a9362
:auto-6511ae28
:auto-98e6f251
:auto-82487b15
:auto-e3f195f4
:auto-7ff2762f
:auto-2ec04bb6
:feat/connectivity-tester-release-workflow-admin
:auto-558e1368
:auto-be0b9518
:auto-525201f6
:auto-418ecdd8
:auto-418ecdd8-9436806f
:auto-8ecc6f04
:auto-9436806f
:auto-test-74229c1
:auto-703e0198
:release/0.1.0
:feat/monitoring-prod-watchdog
:fix/gitea-act-runner-cache-server
:docs/hermes_infra-gateway-lag-recovery
:docs/hermes_infra-telegram-mapping
:fix/hermes-gateway-memory-pids
:fix/hermes-gateway-healthcheck
:feat/docker-socket-proxy-readonly
:docs/architecture-diagram-cleanup
:docs/architecture-diagram
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3a52bb97e1 | feat(connectivity): verify Hermes internet and NFS | ||
|
|
74c3b435e8 |
feat(connectivity): deploy tester on all swarm nodes (#48)
## Summary Add the connectivity_tester deployment from the Hermes release branch to admin/master. ## Changes - add the connectivity_tester stack and Brencher integration - dispatch the multi-arch image build to Gitea Actions - publish the image to the admin registry - deploy with registry authentication forwarded to Swarm nodes ## Verification - Actions build completed successfully - Swarm runtime verified on odroid, xu4, and rudolf-T - 3/3 tasks running Source branch: `hermes/ansible@release/0.1.0-connectivity` --------- Co-authored-by: hermes <hermes@rudolf.keenetic.link> Co-authored-by: brencher_bot <rudolfss13@gmail.com> Reviewed-on: admin/ansible#48 Co-authored-by: Alex Rudenk <rudolfss13+hermes@gmail.com> |
||
|
|
82487b1565 |
[ready-to-test] feat(hermes): use NFS volumes on odroid SSD (#46)
release / build (push) Failing after 23s
## Summary Configure the `hermes` stack persistent volumes as NFS-backed volumes on the SSD of `odroid`. Changed only: - `hermes_gateway_data` → `192.168.1.96:/root/volumes/hermes/gateway` - `hermes_agent_data` → `192.168.1.96:/root/volumes/hermes/agent` `hermes_squid` is not included. ## Validation - `docker stack config -c poc/hermes/stack-compose.yml` succeeds. - `git diff --check` succeeds. - Diff contains only `poc/hermes/stack-compose.yml`. ## Deployment prerequisite Before deploy, prepare the NFS export and copy the existing local volumes according to issue #45: admin/ansible#45 Do not deploy before verifying the copy, NFS access from `rudolf-T`, and the absence of concurrent writes to the old volumes. ## Tracking Resolves the storage work in admin/ansible#45: admin/ansible#45 Co-authored-by: hermes <rudolfss13@gmail.com> Reviewed-on: admin/ansible#46 Co-authored-by: hermes <rudolfss13+hermes@gmail.com> |
||
|
|
e3f195f4bc |
fix(connectivity_tester): publish to admin registry (#42)
release / build (push) Failing after 20s
Publishes the connectivity_tester image to the admin repository registry where the workflow runs. Tracking: admin/ansible#16 Reviewed-on: admin/ansible#42 Co-authored-by: Alex Rudenk <rudolfss13+hermes@gmail.com> |
||
|
|
9436806f78 |
ci: build connectivity tester from admin branches (#41)
release / build (push) Failing after 32s
## Changes - add the release workflow to `admin/ansible` - trigger builds for `auto-*` branches in the administrative repository - build and push `gitea.rudolf.keenetic.link/hermes/connectivity_tester` - update the stack image to the same registry target This is required by Brencher after auto-branches are pushed to `gitea_admin`. Reviewed-on: admin/ansible#41 Co-authored-by: Alex Rudenk <rudolfss13+hermes@gmail.com> |
||
|
|
ff08677d12 |
feat(connectivity_tester): new stack for synthetic connectivity checks (#39)
## Summary New Swarm stack `connectivity_tester` — a long-running Python service that probes the cluster's connectivity every 10 minutes and logs structured PASS/FAIL per check. ## What's in the box - `connectivity_tester/Dockerfile` — python:3.12-slim base - `connectivity_tester/app.py` — main loop: read env, run round, sleep, repeat - `connectivity_tester/endpoints.py` — `EXPECTED_INTERNAL`, `EXPECTED_EXTERNAL`, `NEGATIVE` lists - `connectivity_tester/stack-compose.yml` — Swarm stack with `mode: global` (one task per node) ## Behaviour - One round every `PERIOD_SECONDS` (default 600). - Per round: TCP probes against the three allow-lists + the negative-list. - Negative-list endpoints (cloud metadata services etc.) MUST NOT respond. If they do → log `ERROR LEAK`, container keeps running. - `EXTRA_ENDPOINTS` env var accepts comma-separated `host:port` pairs for ad-hoc checks. ## Placement `mode: global` — Swarm schedules one task on each node. No node pinning. Networks: `hermes_proxy` overlay (for internal DNS resolution) + `default` (for egress via squid). ## Acceptance - After deploy, `docker service logs connectivity_tester_connectivity-tester` shows one round per 10 minutes. - Logs visible at info level in normal conditions (squid, hermes-gateway, etc. all PASS). - Any unexpected reachability or unreachable internal endpoint triggers ERROR-level log within `CHECK_TIMEOUT_S`. ## Related - Org1/hermes-voice-gateway#3 (Path B: build custom voice-gateway) — blocked by Org1/hermes-voice-gateway#5 (connectivity testing container) - Org1/hermes-voice-gateway#5 — this PR resolves the blocker ## Verification checklist - [ ] Stack `connectivity_tester` shows up in `docker stack ls` - [ ] `docker service ps connectivity_tester_connectivity-tester` shows one task per node (odroid, rudolf-T, xu4) - [ ] Logs show first round within 30 seconds of start - [ ] `docker service logs --tail 50 connectivity_tester_connectivity-tester` shows PASS for at least squid:3128 and hermes-gateway:8000 - [ ] Negative-list (169.254.169.254) is unreachable (PASS OK-NEG) --------- Co-authored-by: brencher_bot <rudolfss13@gmail.com> Co-authored-by: hermes <hermes@rudolf.keenetic.link> Reviewed-on: admin/ansible#39 Co-authored-by: hermes <rudolfss13+hermes@gmail.com> |
||
|
|
0998f4bc05 |
feat(hermes_squid): whitelist GitHub Copilot domains (#40)
Mirror of hermes/ansible#6. Allow Hermes to reach GitHub Copilot network endpoints through the corporate squid proxy. Adds a `copilot_domains` ACL covering the domains required by GitHub Copilot (IDE integration and ACP): - `api.github.com` — user/auth and copilot REST endpoints - `copilot-proxy.githubusercontent.com` — completion and chat streaming backend - `*.githubcopilot.com` — newer Copilot control plane - `copilot-telemetry.githubusercontent.com` — older Copilot telemetry - `default.exp-tas.com` — Microsoft AID/Experimentation rollout - `marketplace.visualstudio.com` + `vscode.blob.core.windows.net` — VS Code Marketplace and VSIX downloads - `*.githubassets.com`, `*.vscode-unpkg.net` — asset/CDN hosts used by extension UI `.github.com`, `*.githubusercontent.com`, and the rest of `stt_pkg_domains` were already allowed, so no overlap. Validation: a simple login + completion request from an IDE through `http://squid:3128`. Reviewed-on: admin/ansible#40 Co-authored-by: Alex Rudenk <rudolfss13+hermes@gmail.com> |
||
|
|
f071c7fb43 |
fix(hermes_infra/gitea): add dedicated act_runner cache-server (#34)
## Problem The built-in `act_runner` cache server binds to an interface unreachable from per-job container networks (`container.network: ""` is the default). `ACTIONS_CACHE_URL` ends up pointing at an unreachable IP, and every CI job burns ~9 minutes on: ``` ::warning::Failed to restore: getCacheEntry failed: connect ETIMEDOUT 192.168.112.21:37051 ::warning::Failed to save: reserveCache failed: connect ETIMEDOUT 192.168.112.21:37051 ``` Measured on `admin/brencher#15` (https://gitea.rudolf.keenetic.link/admin/brencher/actions/runs/16): | Job | wall-clock | of which dead air on cache | |---------|------------|----------------------------| | 3.12.3 | 12m 40s | ~9m (271s × 2 timeouts) | | 3.13 | 11m 13s | ~9m | | 3.14 | 11m 17s | ~9m | So even though `python-package.yml` enables `cache: 'pip'`, the cache backend is unreachable, jobs hit `pip install` cold every time, and CI ends up ~4x slower than necessary. ## Fix Run a dedicated `act_runner cache-server` container on the same overlay network as the runner, and point the runner at it via `external_server`. Shared secret comes from a Docker secret mounted into both containers and read via `ACT_RUNNER_EXTERNAL_SECRET_FILE`. ## Changes - `poc/gitea/stack-compose.yml`: - New `cache_server` service (`command: ["cache-server"]`) on the `gitea` overlay with a `cache_data` named volume. - `runner` and `cache_server` both mount the external Docker secret `gitea_cache_secret` at `/run/secrets/gitea_cache_secret` and set `ACT_RUNNER_EXTERNAL_SECRET_FILE` accordingly. - Top-level `secrets:` block declares `gitea_cache_secret` as `external: true`, with the create command commented inline next to it. - `poc/gitea/config.yaml`: point `cache.{host,port,external_server}` at `gitea_cache_server:37051`; `external_secret` left empty because the secret is loaded from the file. ## Verification - Both YAMLs parse (PyYAML). - Service name `gitea_cache_server` resolves on the `gitea` overlay via Swarm service-name DNS. - `docker secret inspect gitea_cache_secret` works after the one-time create step. ## Deploy notes 1. On the swarm manager, create the secret once: ``` echo -n "$(openssl rand -hex 32)" | docker secret create gitea_cache_secret - ``` 2. `docker stack deploy -c stack-compose.yml gitea`. 3. Re-run a CI job and confirm cache `restore`/`save` log lines no longer ETIMEDOUT. ## Related - Related: admin/brencher#15 — `python-package.yml` enables `cache: 'pip'` and was timing out for the reason this PR fixes. - Refs: gitea/actions/runs/16 jobs #248/249/250. ## Supersedes - Supersedes: admin/ansible#33 — first attempt was based on a stale `origin/master` and accidentally included `hermes_squid/` changes from the actual upstream merge. The current branch is rebased on `admin/master` (`c3b7128`) and only touches `poc/gitea/`.Reviewed-on: http://192.168.1.96/admin/ansible/pulls/34 Reviewed-by: admin <rudolfss13@gmail.com> Co-authored-by: hermes <rudolfss13+hermes@gmail.com> |
||
|
|
c3b7128006 |
hermes_squid: open docker_socket_proxy to full read-only scope (#4)
Flip CONTAINERS/IMAGES/NETWORKS/VOLUMES/SERVICES/TASKS/NODES from 0 to 1 so hermes-agent can inspect running stacks (read container logs, enumerate swarm services) without going through each node's shell. POST stays 0 — no container creation, no image builds, no exec into running containers. Anything that mutates state remains blocked at the proxy level. --------- Co-authored-by: admin <rudolfss13@gmail.com> Co-authored-by: hermes <hermes@rudolf.keenetic.link> Reviewed-on: http://192.168.1.96/admin/ansible/pulls/4 Co-authored-by: hermes <rudolfss13+hermes@gmail.com> |
||
|
|
0c95d8e1ee |
feat(hermes_squid): enable local STT via faster-whisper (runtime install) (#21)
## What Wires up local STT (faster-whisper) into the hermes-gateway container so voice messages on Telegram transcribe without depending on external services. Closes #19. Supersedes the closed #20 (which used a custom Dockerfile-hermes; this version uses a bind-mounted entrypoint script instead since pypi.org is now whitelisted). ## Changes 1. **`hermes_squid/squid.conf`** — expands the egress whitelist: - new `stt_pkg_domains` for `pypi.org`, `files.pythonhosted.org`, `huggingface.co`, `github.com`, `models.dev`, `registry.npmjs.org`, `cdn-lfs.huggingface.co` — needed for runtime `pip install faster-whisper` and HF model fetch. - new `telegram_ips` covering AS62041 (91.108.4.0/22, 91.108.8.0/22, 91.108.12.0/22, 91.108.16.0/22, 91.108.20.0/22, 91.108.56.0/22, 149.154.160.0/20, 185.76.151.0/24) as a backstop for the `dstdomain` ACL. squid's `dstdomain` does not match HTTPS CONNECT requests without `ssl_bump`, so the recent log shows 24 of 9083 `api.telegram.org` requests getting 403 in a 5-minute window. - renamed `hermes_network` -> `localnet` (squid default name, no behavior change). 2. **`hermes_squid/hermes-gateway-entrypoint.sh`** (new) — bind-mounted into the container at `/opt/data/hermes-gateway-entrypoint.sh` and set as the service entrypoint. The script: - is idempotent: on first start, runs `apt-get install -y python3-pip ffmpeg` and `pip3 install --break-system-packages faster-whisper`. Subsequent restarts skip install if `faster_whisper` is already importable. - exports `STT_PROVIDER=local`, `STT_LOCAL_MODEL=base`, `STT_LOCAL_DEVICE=cpu`, `STT_LOCAL_COMPUTE_TYPE=int8`. - `exec`s the original gateway command (`gateway run`). 3. **`hermes_squid/stack-compose.yml`** — `hermes-gateway` gets: - `entrypoint: ["/opt/data/hermes-gateway-entrypoint.sh"]` - `command: ["gateway", "run"]` - bind-mount `./hermes-gateway-entrypoint.sh:/opt/data/hermes-gateway-entrypoint.sh:ro` - new env: `STT_PROVIDER`, `STT_LOCAL_MODEL`, `STT_LOCAL_DEVICE`, `STT_LOCAL_COMPUTE_TYPE` No other service in the compose file is touched. ## Why runtime install, not a custom image - `pypi.org` is now whitelisted, so `pip3 install` works at runtime through the same squid. - Avoids maintaining a custom hermes-agent image and a local registry tag. - The upstream `nousresearch/hermes-agent:v2026.8.3` image is kept as-is. ## Deploy notes - brencher apply will trigger `docker service update` for hermes-gateway; the new entrypoint runs the apt+pip install on first start (~1-2 min). - First start also downloads the `base` faster-whisper model (~150 MB) from `huggingface.co` — covered by `stt_pkg_domains`. - squid config is reread on `docker service update` for the squid service; new ACLs apply without a full stack restart. - Subsequent restarts of hermes-gateway skip the install path entirely (idempotent guard). ## Verified locally - `git diff --stat upstream/master` shows only the three files above (+25 entrypoint, +14 squid, +8 compose). - The new worktree is `/opt/data/work/thread-114-Alex-InfraAlexBot/ansible` (per-session working dir). ## Risks - `apt-get update` inside the container needs `deb.debian.org` reachable — already in `allowed_domains`. - `STT_LOCAL_MODEL=base` is ~150 MB and runs on CPU. If latency on Raspberry-class nodes is bad, switch to `tiny`. - `--break-system-packages` is required for PEP 668 (Debian trixie + Python 3.11+). If the upstream image switches to PEP 668 strictly, this flag is the right escape hatch. ## Related - #19 — investigation that found missing faster-whisper + squid deny pattern - #20 — superseded (used Dockerfile-hermes instead of runtime install) --------- Co-authored-by: hermes <hermes@rudolf.keenetic.link> Reviewed-on: http://192.168.1.96/admin/ansible/pulls/21 Reviewed-by: admin <rudolfss13@gmail.com> Co-authored-by: hermes <rudolfss13+hermes@gmail.com> |