diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 0000000..4bd565c --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,128 @@ +# Architecture diagram + +> Diagram: [`mermaid.md`](./mermaid.md) — keep this updated. + +## When user asks to update the diagram + +1. **Refresh live state from `docker_socket_proxy`** (read-only, GET-only): + ```bash + PROXY=http://docker_socket_proxy:2375 + + # Stacks (via com.docker.stack.namespace label on services) + curl -sS $PROXY/services | python3 -c " + import json,sys + from collections import Counter + c = Counter() + for s in json.load(sys.stdin): + ns = s['Spec'].get('Labels',{}).get('com.docker.stack.namespace','(none)') + c[ns] += 1 + for ns, n in sorted(c.items()): + print(f'{ns:30} {n} services')" + + # Replicas per service + curl -sS $PROXY/services | python3 -c " + import json,sys + for s in json.load(sys.stdin): + spec = s['Spec'] + mode = spec.get('Mode',{}).get('Replicated',{}).get('Replicas','?') + print(f'{spec[\"Name\"]:50} replicas={mode}')" + + # Volumes summary + curl -sS $PROXY/volumes | python3 -c " + import json,sys,urllib.request + vols = json.load(sys.stdin)['Volumes'] + running = set() + for c in json.loads(urllib.request.urlopen('$PROXY/containers/json').read()): + if c.get('State') != 'running': continue + for m in c.get('Mounts', []): + if m.get('Type') == 'volume': + running.add(m['Name']) + print(f'TOTAL: {len(vols)}, mounted: {len(running)}, orphaned: {len(vols)-len(running)}')" + + # Container state histogram (incl. zombie detection) + curl -sS $PROXY/containers/json?all=1 | python3 -c " + import json,sys + from collections import Counter + cs = json.load(sys.stdin) + print(f'TOTAL: {len(cs)}') + for st, n in Counter(c.get('State') for c in cs).most_common(): + print(f' {st}: {n}')" + + # BRENCHER PROFILES (which stacks are managed by brencher) + curl -sS $PROXY/containers/json | python3 -c " + import json,sys + for c in json.load(sys.stdin): + if 'brencher-backend' in (c.get('Names',[''])[0]) and c.get('State') == 'running': + for e in c.get('Config',{}).get('Env',[]): + if e.startswith('PROFILES='): + print(f'BRENCHER PROFILES: {e[9:]}') + break" + + # Images in use + curl -sS $PROXY/containers/json | python3 -c " + import json,sys + from collections import Counter + c = Counter() + for ct in json.load(sys.stdin): + if ct.get('State') != 'running': continue + c[ct.get('Image','?').split('@')[0]] += 1 + for img, n in c.most_common(): + print(f' {n}x {img}')" + + # Swarm nodes + curl -sS $PROXY/nodes | python3 -c " + import json,sys + for n in json.load(sys.stdin): + d = n.get('Description',{}) + print(f'{d.get(\"Hostname\"):10} role={n[\"Spec\"][\"Role\"]} addr={n[\"Status\"][\"Addr\"]}')" + + # Overlay network count + curl -sS $PROXY/networks | python3 -c " + import json,sys + n = sum(1 for x in json.load(sys.stdin) if x.get('Scope')=='swarm') + print(f'overlay networks: {n}')" + ``` + +2. **Compare against `mermaid.md`**. Update what changed: + - New stacks → add under BRENCHER_DEPLOYED (if PROFILES has them) or STANDALONE + - Stack moved to zombie (was running, now Replicas=0) → move to ZOMBIE swimlane + - Stack came back to life → move back + - New ephemeral job count (rough — only matters if asked) + - Any new cross-stack network names → update proxy_egress label + - Volumes: only mention if the active count moves significantly + +3. **Don't bother** (waste of time, low value): + - Per-pod ephemeral names — just count total + - Individual container IPs (no SVG variant now) + - Per-volume mount details + - Network IP ranges for individual services + +4. **Endpoints that return 403 (skip these)**: + - `GET /configs` — needs POST=1 + - `GET /secrets` — needs POST=1 + - `GET /info` — needs POST=1 + These exist but aren't readable through the proxy. Known secrets are documented in `mermaid.md` and don't need refreshing. + +5. **Push** (NO force-push, just a normal commit on top): + ```bash + cd /opt/data/work/diagram-cleanup # or wherever this worktree is + # edit mermaid.md + git add docs/architecture/mermaid.md + git commit -m "docs: refresh prod map (snapshot YYYY-MM-DD)" + git push origin docs/architecture-diagram-cleanup + ``` + Then open or update the PR against `admin/ansible:master`. + +## What's intentionally NOT in the diagram + +- Container IPs (was in old standalone-adaptive.svg, removed because too noisy) +- Full list of 119 anonymous volumes (just count) +- Per-mount details for each container +- Internal stack service-to-service traffic (only ingress/egress/edge points shown) + +## Known limitations / don't fix unless asked + +- `endpoint.Ports` for `gitea_server` is empty but ingress still routes — legacy/race condition, leave alone +- `nginx_nginx` service is Replicas=0 but overlay network `nginx_ingress` stays alive — orphan, leave alone +- `act_runner` spawns ephemeral job containers (random names like `romantic_wozniak`) on every CI run — they accumulate as exited containers, that's normal +- `brencher_tmp` volume gets re-populated by every `docker stack deploy` — that's how brencher works diff --git a/docs/architecture/mermaid.md b/docs/architecture/mermaid.md new file mode 100644 index 0000000..20ede16 --- /dev/null +++ b/docs/architecture/mermaid.md @@ -0,0 +1,253 @@ +# Architecture diagram — full prod map + +> **Snapshot:** 2026-08-27 · sources: `docker_socket_proxy` + gitea admin API +> +> Single Mermaid view — high-level topology of all prod elements. +> For per-node detail (IPs, container names), run the update commands in `README.md` +> against a fresh snapshot. + +```mermaid +flowchart LR + %% ==================== EXTERNAL ==================== + subgraph EXT[" "] + direction TB + Client["Client
tailscale"]:::client + Keenetic["Keenetic
DNS · TLS"]:::external + Tailscale["Tailscale
subnet router"]:::external + lan["lan
192.168.1.0/24"]:::external + + Client -->|https| Tailscale + Tailscale --> Keenetic + Keenetic -->|TLS term| lan + end + + %% ==================== INGRESS / EGRESS ==================== + subgraph IO[" "] + direction TB + ingress["ingress
10.0.0.0/24"]:::egress + proxy_egress["squid_egress
hermes_proxy
admin"]:::egress + nginx_nets["nginx_ingress
nginx_default
(DEAD)"]:::egress + + lan -.->|egress| ingress + end + + %% ==================== STACKS (deployed via brencher) ==================== + subgraph BRENCHER_DEPLOYED["via brencher (PROFILES env)"] + direction TB + b_gitea["gitea"]:::stack + b_hermes["hermes"]:::stack + b_hermes_infra["hermes_infra"]:::stack + b_immich["immich"]:::stack + b_gmail_mcp["gmail_mcp"]:::stack + b_torrserv["torrserv_proxy"]:::stack + b_brencher2["brencher2
(deploy tool)"]:::deploy + end + + ingress --> b_gitea + ingress --> b_hermes + ingress --> b_hermes_infra + ingress --> b_immich + ingress --> b_gmail_mcp + ingress --> b_torrserv + + proxy_egress -.-> b_hermes_infra + + b_brencher2 -.->|deploys| b_gitea + b_brencher2 -.->|deploys| b_hermes + b_brencher2 -.->|deploys| b_immich + b_brencher2 -.->|deploys| b_hermes_infra + b_brencher2 -.->|deploys| b_gmail_mcp + b_brencher2 -.->|deploys| b_torrserv + + %% ==================== STACKS (standalone, not via brencher) ==================== + subgraph STANDALONE["standalone (compose или ручной deploy)"] + direction TB + s_dsp["docker_socket_proxy"]:::stack + s_lampa["lampa-jackett
(compose)"]:::stack + s_photosHelper["photosHelper
(DEAD · Replicas: 0)"]:::dead + s_nginx["nginx
(DEAD · Replicas: 0)"]:::dead + s_registry["registry
(DEAD · Replicas: 0)"]:::dead + end + + ingress --> s_dsp + ingress --> s_lampa + + %% ==================== ZOMBIE STACKS (volumes only, no services) ==================== + subgraph ZOMBIE["zombie stacks (volumes orphaned)"] + direction TB + z_nextcloud["nextcloud
(4 volumes)"]:::dead + z_portainer["portainer
(1 volume)"]:::dead + z_swarmpit["swarmpit
(1 volume)"]:::dead + z_sstp["sstp · softether
(4 volumes)"]:::dead + z_test["test
(1 volume)"]:::dead + z_torr_cache["torr_cache
(1 volume)"]:::dead + z_beszel["beszel-poc
(2 volumes)"]:::dead + end + + %% ==================== EPHEMERAL (gitea act_runner jobs) ==================== + subgraph EPHEMERAL["gitea act_runner ephemeral jobs"] + direction TB + eph_jobs["~48 exited
ephemeral job pods
(random names)"]:::ephemeral + end + + b_gitea -.->|spawns| eph_jobs + + %% ==================== INTERNAL NETS (one per stack) ==================== + subgraph INTERNAL["internal nets (12)"] + direction TB + gitea_net["gitea_gitea"]:::internal + hermes_net["hermes_hermes"]:::internal + hi_net["hermes_infra_default"]:::internal + immich_net["immich_default"]:::internal + gmail_net["gmail_mcp_default"]:::internal + torr_net["torrserv_proxy_default"]:::internal + brencher2_nets["brencher2-net +
brencher2_default"]:::internal + dsp_net["dsp_default"]:::internal + lampa_net["lampa_default"]:::internal + + b_gitea --> gitea_net + b_hermes --> hermes_net + b_hermes_infra --> hi_net + b_immich --> immich_net + b_gmail_mcp --> gmail_net + b_torrserv --> torr_net + b_brencher2 --> brencher2_nets + s_dsp --> dsp_net + s_lampa --> lampa_net + end + + %% ==================== VOLUMES (153 total) ==================== + subgraph VOLS["volumes (153 total, 18 mounted · 135 orphaned)"] + direction TB + vol_active["active: gitea_data · immich_* ·
hermes_*_data · brencher_tmp ·
docker_socket_proxy_*_data ·
torrserv_proxy_data · lampa_*"]:::volumes + vol_orphaned["orphaned: 119 (no stack) ·
nextcloud · portainer · sstp ·
swarmpit · test · torr_cache ·
beszel-poc · immich_postgres"]:::dead + end + + %% ==================== SECRETS / CONFIGS (403 from socket proxy) ==================== + subgraph SECRETS["secrets & configs (read-only proxy: 403)"] + direction TB + sec_known["known to exist in gitea app.ini:
LFS_JWT_SECRET · JWT_SECRET ·
SECRET_KEY · INTERNAL_TOKEN"]:::secret + sec_hermes["hermes-agent runtime secrets
(gmail creds · telegram token)"]:::secret + end + + %% ==================== STYLING ==================== + classDef client fill:#0f172a,stroke:#a78bfa,color:#ddd6fe + classDef external fill:#1e293b,stroke:#94a3b8,color:#cbd5e1 + classDef egress fill:#052e16,stroke:#22c55e,color:#bbf7d0 + classDef stack fill:#0c4a6e,stroke:#22d3ee,color:#cffafe + classDef dead fill:#1f1717,stroke:#fb7185,color:#fecaca,stroke-dasharray: 4 2 + classDef deploy fill:#422006,stroke:#fbbf24,color:#fde68a + classDef ephemeral fill:#1e1b4b,stroke:#818cf8,color:#c7d2fe,stroke-dasharray: 2 2 + classDef internal fill:#1e1b4b,stroke:#818cf8,color:#c7d2fe + classDef volumes fill:#0f172a,stroke:#475569,color:#cbd5e1 + classDef secret fill:#0f172a,stroke:#f59e0b,color:#fde68a,stroke-dasharray: 3 2 +``` + +## Полная картина прода + +### 🟢 Running stacks (12) + +**Через `brencher` (PROFILES env):** +1. **gitea** — gitea_server + gitea_runner (act_runner для CI) +2. **hermes** — hermes-agent + hermes-gateway +3. **hermes_infra** — squid + sarg + hermes-ui + hermes-gateway + docker_socket_proxy + brencher-state-proxy +4. **immich** — server + database + redis + machine-learning +5. **gmail_mcp** — gmail-mcp-server + telegram-mcp-server +6. **torrserv_proxy** — reverseproxy + fileserver +7. **brencher2** — второй deploy tool (сам себя не деплоит) + +**Standalone (не через brencher):** +8. **docker_socket_proxy** — standalone, для hermes-gateway +9. **lampa-jackett** — docker-compose, port 9117 + +**DEAD services (но стек определён):** +10. **photosHelper** (Replicas: 0, 1 volume) +11. **nginx** (Replicas: 0, 2 services, 5 volumes) +12. **registry** (Replicas: 0, 1 volume) + +### 🟡 Zombie stacks (volumes, но 0 running services) + +Volumes остались после того как сервисы удалили. 4–5 volumes на стек: + +- **nextcloud** (4 volumes · config + данные) +- **sstp** (4 volumes · SoftEther VPN, конфиг + logs) +- **portainer** (1 volume · swarm UI) +- **swarmpit** (1 volume · swarm UI) +- **test** (1 volume) +- **torr_cache** (1 volume) +- **beszel-poc** (2 volumes · monitoring POC) + +### 🟣 Ephemeral: gitea act_runner job pods + +Когда CI job запускается в gitea, `act_runner` создаёт **ephemeral контейнер** со случайным именем (`romantic_wozniak`, `compassionate_goodall` и т.п.). После завершения контейнер остаётся в `exited` state. На текущий момент ~48 таких exited контейнеров. + +Реестр: image `ghcr.io/immich-app/runner-images:ubuntu-latest` (определён в `poc/gitea/config.yaml`). + +### 🔴 Volumes (153 total) + +- **18 mounted** в running контейнеры +- **135 orphaned** (либо от zombie-стеков, либо anonymous) + +Основные используемые: +- `gitea_data` — критичные данные gitea (NFS на 192.168.1.96:/root/volumes/gitea) +- `immich_*` — uploads, model-cache, postgres +- `hermes_infra_*_data`, `hermes_hermes_*_data` +- `brencher_tmp`, `brencher2_tmp` — git clones +- `lampa_jackett_config` +- `torrserv_proxy_data` + +### 🔐 Secrets & configs + +`/configs` и `/secrets` через socket-proxy **403** (требуется POST:1). Известные по другим источникам: + +**В gitea `/data/gitea/conf/app.ini`:** +- `LFS_JWT_SECRET` +- `JWT_SECRET` (oauth2) +- `SECRET_KEY` +- `INTERNAL_TOKEN` + +**В hermes-agent runtime:** +- Gmail API creds (для gmail_mcp server) +- Telegram bot token (для telegram_mcp server) + +**Swarm configs** (из `poc/gitea/stack-compose.yml`): +- `runner_config` — `config.yaml` для act_runner + +### 🗑️ Dead-overlay сети (сеть есть, контейнеров нет) + +- `nginx_ingress` (10.0.3.0/24) — для nginx-proxy, но nginx Replicas=0 +- `nginx_default` (10.0.4.0/24) — то же + +Сети **живые** в swarm mesh, но не используются. + +## Что НЕ показывает эта диаграмма + +- Конкретные IP контейнеров (см. README.md — собираются через `docker_socket_proxy`) +- Cross-stack сети в деталях (см. SVG) +- Точные timestamps exited containers +- Полный список 119 orphaned volumes без stack'а + +## Как обновить + +Запусти в этом окружении: + +```bash +# Stacks & services +curl http://docker_socket_proxy:2375/services | jq -r '.[] | .Spec.Labels["com.docker.stack.namespace"]' | sort -u + +# Volumes per stack +curl http://docker_socket_proxy:2375/volumes | jq -r '.Volumes[] | "\(.Name) \(.Labels["com.docker.stack_namespace"] // "no-stack")"' + +# Ephemeral containers (exited) +curl http://docker_socket_proxy:2375/containers/json?all=1 | jq '[.[] | select(.State=="exited")] | length' + +# BRENCHER profiles (env var) +curl http://docker_socket_proxy:2375/containers/brencher_brencher-backend* 2>/dev/null +``` + +## Связанные + +- `README.md` — команды обновления этой диаграммы +- `admin/ansible#5` — re-deploy hermes_squid +- `admin/ansible#6` — pre-create socket_proxy overlay +- `admin/ansible#7` — gitea stack deploy / app.ini fix