docs: remove SVG variant, keep only Mermaid diagram #22

Open
hermes wants to merge 15 commits from hermes/ansible:docs/architecture-diagram-cleanup into master
2 changed files with 381 additions and 0 deletions
+128
View File
@@ -0,0 +1,128 @@
# Architecture diagram
> Diagram: [`mermaid.md`](./mermaid.md) — keep this updated.
## When user asks to update the diagram
1. **Refresh live state from `docker_socket_proxy`** (read-only, GET-only):
```bash
PROXY=http://docker_socket_proxy:2375
# Stacks (via com.docker.stack.namespace label on services)
curl -sS $PROXY/services | python3 -c "
import json,sys
from collections import Counter
c = Counter()
for s in json.load(sys.stdin):
ns = s['Spec'].get('Labels',{}).get('com.docker.stack.namespace','(none)')
c[ns] += 1
for ns, n in sorted(c.items()):
print(f'{ns:30} {n} services')"
# Replicas per service
curl -sS $PROXY/services | python3 -c "
import json,sys
for s in json.load(sys.stdin):
spec = s['Spec']
mode = spec.get('Mode',{}).get('Replicated',{}).get('Replicas','?')
print(f'{spec[\"Name\"]:50} replicas={mode}')"
# Volumes summary
curl -sS $PROXY/volumes | python3 -c "
import json,sys,urllib.request
vols = json.load(sys.stdin)['Volumes']
running = set()
for c in json.loads(urllib.request.urlopen('$PROXY/containers/json').read()):
if c.get('State') != 'running': continue
for m in c.get('Mounts', []):
if m.get('Type') == 'volume':
running.add(m['Name'])
print(f'TOTAL: {len(vols)}, mounted: {len(running)}, orphaned: {len(vols)-len(running)}')"
# Container state histogram (incl. zombie detection)
curl -sS $PROXY/containers/json?all=1 | python3 -c "
import json,sys
from collections import Counter
cs = json.load(sys.stdin)
print(f'TOTAL: {len(cs)}')
for st, n in Counter(c.get('State') for c in cs).most_common():
print(f' {st}: {n}')"
# BRENCHER PROFILES (which stacks are managed by brencher)
curl -sS $PROXY/containers/json | python3 -c "
import json,sys
for c in json.load(sys.stdin):
if 'brencher-backend' in (c.get('Names',[''])[0]) and c.get('State') == 'running':
for e in c.get('Config',{}).get('Env',[]):
if e.startswith('PROFILES='):
print(f'BRENCHER PROFILES: {e[9:]}')
break"
# Images in use
curl -sS $PROXY/containers/json | python3 -c "
import json,sys
from collections import Counter
c = Counter()
for ct in json.load(sys.stdin):
if ct.get('State') != 'running': continue
c[ct.get('Image','?').split('@')[0]] += 1
for img, n in c.most_common():
print(f' {n}x {img}')"
# Swarm nodes
curl -sS $PROXY/nodes | python3 -c "
import json,sys
for n in json.load(sys.stdin):
d = n.get('Description',{})
print(f'{d.get(\"Hostname\"):10} role={n[\"Spec\"][\"Role\"]} addr={n[\"Status\"][\"Addr\"]}')"
# Overlay network count
curl -sS $PROXY/networks | python3 -c "
import json,sys
n = sum(1 for x in json.load(sys.stdin) if x.get('Scope')=='swarm')
print(f'overlay networks: {n}')"
```
2. **Compare against `mermaid.md`**. Update what changed:
- New stacks → add under BRENCHER_DEPLOYED (if PROFILES has them) or STANDALONE
- Stack moved to zombie (was running, now Replicas=0) → move to ZOMBIE swimlane
- Stack came back to life → move back
- New ephemeral job count (rough — only matters if asked)
- Any new cross-stack network names → update proxy_egress label
- Volumes: only mention if the active count moves significantly
3. **Don't bother** (waste of time, low value):
- Per-pod ephemeral names — just count total
- Individual container IPs (no SVG variant now)
- Per-volume mount details
- Network IP ranges for individual services
4. **Endpoints that return 403 (skip these)**:
- `GET /configs` — needs POST=1
- `GET /secrets` — needs POST=1
- `GET /info` — needs POST=1
These exist but aren't readable through the proxy. Known secrets are documented in `mermaid.md` and don't need refreshing.
5. **Push** (NO force-push, just a normal commit on top):
```bash
cd /opt/data/work/diagram-cleanup # or wherever this worktree is
# edit mermaid.md
git add docs/architecture/mermaid.md
git commit -m "docs: refresh prod map (snapshot YYYY-MM-DD)"
git push origin docs/architecture-diagram-cleanup
```
Then open or update the PR against `admin/ansible:master`.
## What's intentionally NOT in the diagram
- Container IPs (was in old standalone-adaptive.svg, removed because too noisy)
- Full list of 119 anonymous volumes (just count)
- Per-mount details for each container
- Internal stack service-to-service traffic (only ingress/egress/edge points shown)
## Known limitations / don't fix unless asked
- `endpoint.Ports` for `gitea_server` is empty but ingress still routes — legacy/race condition, leave alone
- `nginx_nginx` service is Replicas=0 but overlay network `nginx_ingress` stays alive — orphan, leave alone
- `act_runner` spawns ephemeral job containers (random names like `romantic_wozniak`) on every CI run — they accumulate as exited containers, that's normal
- `brencher_tmp` volume gets re-populated by every `docker stack deploy` — that's how brencher works
+253
View File
@@ -0,0 +1,253 @@
# Architecture diagram — full prod map
> **Snapshot:** 2026-08-27 · sources: `docker_socket_proxy` + gitea admin API
>
> Single Mermaid view — high-level topology of all prod elements.
> For per-node detail (IPs, container names), run the update commands in `README.md`
> against a fresh snapshot.
```mermaid
flowchart LR
%% ==================== EXTERNAL ====================
subgraph EXT[" "]
direction TB
Client["Client<br/>tailscale"]:::client
Keenetic["Keenetic<br/>DNS · TLS"]:::external
Tailscale["Tailscale<br/>subnet router"]:::external
lan["lan<br/>192.168.1.0/24"]:::external
Client -->|https| Tailscale
Tailscale --> Keenetic
Keenetic -->|TLS term| lan
end
%% ==================== INGRESS / EGRESS ====================
subgraph IO[" "]
direction TB
ingress["ingress<br/>10.0.0.0/24"]:::egress
proxy_egress["squid_egress<br/>hermes_proxy<br/>admin"]:::egress
nginx_nets["nginx_ingress<br/>nginx_default<br/>(DEAD)"]:::egress
lan -.->|egress| ingress
end
%% ==================== STACKS (deployed via brencher) ====================
subgraph BRENCHER_DEPLOYED["via brencher (PROFILES env)"]
direction TB
b_gitea["gitea"]:::stack
b_hermes["hermes"]:::stack
b_hermes_infra["hermes_infra"]:::stack
b_immich["immich"]:::stack
b_gmail_mcp["gmail_mcp"]:::stack
b_torrserv["torrserv_proxy"]:::stack
b_brencher2["brencher2<br/>(deploy tool)"]:::deploy
end
ingress --> b_gitea
ingress --> b_hermes
ingress --> b_hermes_infra
ingress --> b_immich
ingress --> b_gmail_mcp
ingress --> b_torrserv
proxy_egress -.-> b_hermes_infra
b_brencher2 -.->|deploys| b_gitea
b_brencher2 -.->|deploys| b_hermes
b_brencher2 -.->|deploys| b_immich
b_brencher2 -.->|deploys| b_hermes_infra
b_brencher2 -.->|deploys| b_gmail_mcp
b_brencher2 -.->|deploys| b_torrserv
%% ==================== STACKS (standalone, not via brencher) ====================
subgraph STANDALONE["standalone (compose или ручной deploy)"]
direction TB
s_dsp["docker_socket_proxy"]:::stack
s_lampa["lampa-jackett<br/>(compose)"]:::stack
s_photosHelper["photosHelper<br/>(DEAD · Replicas: 0)"]:::dead
s_nginx["nginx<br/>(DEAD · Replicas: 0)"]:::dead
s_registry["registry<br/>(DEAD · Replicas: 0)"]:::dead
end
ingress --> s_dsp
ingress --> s_lampa
%% ==================== ZOMBIE STACKS (volumes only, no services) ====================
subgraph ZOMBIE["zombie stacks (volumes orphaned)"]
direction TB
z_nextcloud["nextcloud<br/>(4 volumes)"]:::dead
z_portainer["portainer<br/>(1 volume)"]:::dead
z_swarmpit["swarmpit<br/>(1 volume)"]:::dead
z_sstp["sstp · softether<br/>(4 volumes)"]:::dead
z_test["test<br/>(1 volume)"]:::dead
z_torr_cache["torr_cache<br/>(1 volume)"]:::dead
z_beszel["beszel-poc<br/>(2 volumes)"]:::dead
end
%% ==================== EPHEMERAL (gitea act_runner jobs) ====================
subgraph EPHEMERAL["gitea act_runner ephemeral jobs"]
direction TB
eph_jobs["~48 exited<br/>ephemeral job pods<br/>(random names)"]:::ephemeral
end
b_gitea -.->|spawns| eph_jobs
%% ==================== INTERNAL NETS (one per stack) ====================
subgraph INTERNAL["internal nets (12)"]
direction TB
gitea_net["gitea_gitea"]:::internal
hermes_net["hermes_hermes"]:::internal
hi_net["hermes_infra_default"]:::internal
immich_net["immich_default"]:::internal
gmail_net["gmail_mcp_default"]:::internal
torr_net["torrserv_proxy_default"]:::internal
brencher2_nets["brencher2-net +<br/>brencher2_default"]:::internal
dsp_net["dsp_default"]:::internal
lampa_net["lampa_default"]:::internal
b_gitea --> gitea_net
b_hermes --> hermes_net
b_hermes_infra --> hi_net
b_immich --> immich_net
b_gmail_mcp --> gmail_net
b_torrserv --> torr_net
b_brencher2 --> brencher2_nets
s_dsp --> dsp_net
s_lampa --> lampa_net
end
%% ==================== VOLUMES (153 total) ====================
subgraph VOLS["volumes (153 total, 18 mounted · 135 orphaned)"]
direction TB
vol_active["active: gitea_data · immich_* ·<br/>hermes_*_data · brencher_tmp ·<br/>docker_socket_proxy_*_data ·<br/>torrserv_proxy_data · lampa_*"]:::volumes
vol_orphaned["orphaned: 119 (no stack) ·<br/>nextcloud · portainer · sstp ·<br/>swarmpit · test · torr_cache ·<br/>beszel-poc · immich_postgres"]:::dead
end
%% ==================== SECRETS / CONFIGS (403 from socket proxy) ====================
subgraph SECRETS["secrets & configs (read-only proxy: 403)"]
direction TB
sec_known["known to exist in gitea app.ini:<br/>LFS_JWT_SECRET · JWT_SECRET ·<br/>SECRET_KEY · INTERNAL_TOKEN"]:::secret
sec_hermes["hermes-agent runtime secrets<br/>(gmail creds · telegram token)"]:::secret
end
%% ==================== STYLING ====================
classDef client fill:#0f172a,stroke:#a78bfa,color:#ddd6fe
classDef external fill:#1e293b,stroke:#94a3b8,color:#cbd5e1
classDef egress fill:#052e16,stroke:#22c55e,color:#bbf7d0
classDef stack fill:#0c4a6e,stroke:#22d3ee,color:#cffafe
classDef dead fill:#1f1717,stroke:#fb7185,color:#fecaca,stroke-dasharray: 4 2
classDef deploy fill:#422006,stroke:#fbbf24,color:#fde68a
classDef ephemeral fill:#1e1b4b,stroke:#818cf8,color:#c7d2fe,stroke-dasharray: 2 2
classDef internal fill:#1e1b4b,stroke:#818cf8,color:#c7d2fe
classDef volumes fill:#0f172a,stroke:#475569,color:#cbd5e1
classDef secret fill:#0f172a,stroke:#f59e0b,color:#fde68a,stroke-dasharray: 3 2
```
## Полная картина прода
### 🟢 Running stacks (12)
**Через `brencher` (PROFILES env):**
1. **gitea** — gitea_server + gitea_runner (act_runner для CI)
2. **hermes** — hermes-agent + hermes-gateway
3. **hermes_infra** — squid + sarg + hermes-ui + hermes-gateway + docker_socket_proxy + brencher-state-proxy
4. **immich** — server + database + redis + machine-learning
5. **gmail_mcp** — gmail-mcp-server + telegram-mcp-server
6. **torrserv_proxy** — reverseproxy + fileserver
7. **brencher2** — второй deploy tool (сам себя не деплоит)
**Standalone (не через brencher):**
8. **docker_socket_proxy** — standalone, для hermes-gateway
9. **lampa-jackett** — docker-compose, port 9117
**DEAD services (но стек определён):**
10. **photosHelper** (Replicas: 0, 1 volume)
11. **nginx** (Replicas: 0, 2 services, 5 volumes)
12. **registry** (Replicas: 0, 1 volume)
### 🟡 Zombie stacks (volumes, но 0 running services)
Volumes остались после того как сервисы удалили. 4–5 volumes на стек:
- **nextcloud** (4 volumes · config + данные)
- **sstp** (4 volumes · SoftEther VPN, конфиг + logs)
- **portainer** (1 volume · swarm UI)
- **swarmpit** (1 volume · swarm UI)
- **test** (1 volume)
- **torr_cache** (1 volume)
- **beszel-poc** (2 volumes · monitoring POC)
### 🟣 Ephemeral: gitea act_runner job pods
Когда CI job запускается в gitea, `act_runner` создаёт **ephemeral контейнер** со случайным именем (`romantic_wozniak`, `compassionate_goodall` и т.п.). После завершения контейнер остаётся в `exited` state. На текущий момент ~48 таких exited контейнеров.
Реестр: image `ghcr.io/immich-app/runner-images:ubuntu-latest` (определён в `poc/gitea/config.yaml`).
### 🔴 Volumes (153 total)
- **18 mounted** в running контейнеры
- **135 orphaned** (либо от zombie-стеков, либо anonymous)
Основные используемые:
- `gitea_data` — критичные данные gitea (NFS на 192.168.1.96:/root/volumes/gitea)
- `immich_*` — uploads, model-cache, postgres
- `hermes_infra_*_data`, `hermes_hermes_*_data`
- `brencher_tmp`, `brencher2_tmp` — git clones
- `lampa_jackett_config`
- `torrserv_proxy_data`
### 🔐 Secrets & configs
`/configs` и `/secrets` через socket-proxy **403** (требуется POST:1). Известные по другим источникам:
**В gitea `/data/gitea/conf/app.ini`:**
- `LFS_JWT_SECRET`
- `JWT_SECRET` (oauth2)
- `SECRET_KEY`
- `INTERNAL_TOKEN`
**В hermes-agent runtime:**
- Gmail API creds (для gmail_mcp server)
- Telegram bot token (для telegram_mcp server)
**Swarm configs** (из `poc/gitea/stack-compose.yml`):
- `runner_config` — `config.yaml` для act_runner
### 🗑️ Dead-overlay сети (сеть есть, контейнеров нет)
- `nginx_ingress` (10.0.3.0/24) — для nginx-proxy, но nginx Replicas=0
- `nginx_default` (10.0.4.0/24) — то же
Сети **живые** в swarm mesh, но не используются.
## Что НЕ показывает эта диаграмма
- Конкретные IP контейнеров (см. README.md — собираются через `docker_socket_proxy`)
- Cross-stack сети в деталях (см. SVG)
- Точные timestamps exited containers
- Полный список 119 orphaned volumes без stack'а
## Как обновить
Запусти в этом окружении:
```bash
# Stacks & services
curl http://docker_socket_proxy:2375/services | jq -r '.[] | .Spec.Labels["com.docker.stack.namespace"]' | sort -u
# Volumes per stack
curl http://docker_socket_proxy:2375/volumes | jq -r '.Volumes[] | "\(.Name) \(.Labels["com.docker.stack_namespace"] // "no-stack")"'
# Ephemeral containers (exited)
curl http://docker_socket_proxy:2375/containers/json?all=1 | jq '[.[] | select(.State=="exited")] | length'
# BRENCHER profiles (env var)
curl http://docker_socket_proxy:2375/containers/brencher_brencher-backend* 2>/dev/null
```
## Связанные
- `README.md` — команды обновления этой диаграммы
- `admin/ansible#5` — re-deploy hermes_squid
- `admin/ansible#6` — pre-create socket_proxy overlay
- `admin/ansible#7` — gitea stack deploy / app.ini fix