Compare commits
15
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3e3f9b04bd | ||
|
|
ea077a10a0 | ||
|
|
463da2e48b | ||
|
|
cb285a5461 | ||
|
|
67797c8e7d | ||
|
|
3e68768a22 | ||
|
|
23bb1e8a70 | ||
|
|
8dc466ced3 | ||
|
|
df1b2ce715 | ||
|
|
d500dc04cc | ||
|
|
8672f0db37 | ||
|
|
cd1956aaeb | ||
|
|
e8fa7e8be4 | ||
|
|
7dab6c89ad | ||
|
|
793908ae14 |
@@ -0,0 +1,128 @@
|
||||
# Architecture diagram
|
||||
|
||||
> Diagram: [`mermaid.md`](./mermaid.md) — keep this updated.
|
||||
|
||||
## When user asks to update the diagram
|
||||
|
||||
1. **Refresh live state from `docker_socket_proxy`** (read-only, GET-only):
|
||||
```bash
|
||||
PROXY=http://docker_socket_proxy:2375
|
||||
|
||||
# Stacks (via com.docker.stack.namespace label on services)
|
||||
curl -sS $PROXY/services | python3 -c "
|
||||
import json,sys
|
||||
from collections import Counter
|
||||
c = Counter()
|
||||
for s in json.load(sys.stdin):
|
||||
ns = s['Spec'].get('Labels',{}).get('com.docker.stack.namespace','(none)')
|
||||
c[ns] += 1
|
||||
for ns, n in sorted(c.items()):
|
||||
print(f'{ns:30} {n} services')"
|
||||
|
||||
# Replicas per service
|
||||
curl -sS $PROXY/services | python3 -c "
|
||||
import json,sys
|
||||
for s in json.load(sys.stdin):
|
||||
spec = s['Spec']
|
||||
mode = spec.get('Mode',{}).get('Replicated',{}).get('Replicas','?')
|
||||
print(f'{spec[\"Name\"]:50} replicas={mode}')"
|
||||
|
||||
# Volumes summary
|
||||
curl -sS $PROXY/volumes | python3 -c "
|
||||
import json,sys,urllib.request
|
||||
vols = json.load(sys.stdin)['Volumes']
|
||||
running = set()
|
||||
for c in json.loads(urllib.request.urlopen('$PROXY/containers/json').read()):
|
||||
if c.get('State') != 'running': continue
|
||||
for m in c.get('Mounts', []):
|
||||
if m.get('Type') == 'volume':
|
||||
running.add(m['Name'])
|
||||
print(f'TOTAL: {len(vols)}, mounted: {len(running)}, orphaned: {len(vols)-len(running)}')"
|
||||
|
||||
# Container state histogram (incl. zombie detection)
|
||||
curl -sS $PROXY/containers/json?all=1 | python3 -c "
|
||||
import json,sys
|
||||
from collections import Counter
|
||||
cs = json.load(sys.stdin)
|
||||
print(f'TOTAL: {len(cs)}')
|
||||
for st, n in Counter(c.get('State') for c in cs).most_common():
|
||||
print(f' {st}: {n}')"
|
||||
|
||||
# BRENCHER PROFILES (which stacks are managed by brencher)
|
||||
curl -sS $PROXY/containers/json | python3 -c "
|
||||
import json,sys
|
||||
for c in json.load(sys.stdin):
|
||||
if 'brencher-backend' in (c.get('Names',[''])[0]) and c.get('State') == 'running':
|
||||
for e in c.get('Config',{}).get('Env',[]):
|
||||
if e.startswith('PROFILES='):
|
||||
print(f'BRENCHER PROFILES: {e[9:]}')
|
||||
break"
|
||||
|
||||
# Images in use
|
||||
curl -sS $PROXY/containers/json | python3 -c "
|
||||
import json,sys
|
||||
from collections import Counter
|
||||
c = Counter()
|
||||
for ct in json.load(sys.stdin):
|
||||
if ct.get('State') != 'running': continue
|
||||
c[ct.get('Image','?').split('@')[0]] += 1
|
||||
for img, n in c.most_common():
|
||||
print(f' {n}x {img}')"
|
||||
|
||||
# Swarm nodes
|
||||
curl -sS $PROXY/nodes | python3 -c "
|
||||
import json,sys
|
||||
for n in json.load(sys.stdin):
|
||||
d = n.get('Description',{})
|
||||
print(f'{d.get(\"Hostname\"):10} role={n[\"Spec\"][\"Role\"]} addr={n[\"Status\"][\"Addr\"]}')"
|
||||
|
||||
# Overlay network count
|
||||
curl -sS $PROXY/networks | python3 -c "
|
||||
import json,sys
|
||||
n = sum(1 for x in json.load(sys.stdin) if x.get('Scope')=='swarm')
|
||||
print(f'overlay networks: {n}')"
|
||||
```
|
||||
|
||||
2. **Compare against `mermaid.md`**. Update what changed:
|
||||
- New stacks → add under BRENCHER_DEPLOYED (if PROFILES has them) or STANDALONE
|
||||
- Stack moved to zombie (was running, now Replicas=0) → move to ZOMBIE swimlane
|
||||
- Stack came back to life → move back
|
||||
- New ephemeral job count (rough — only matters if asked)
|
||||
- Any new cross-stack network names → update proxy_egress label
|
||||
- Volumes: only mention if the active count moves significantly
|
||||
|
||||
3. **Don't bother** (waste of time, low value):
|
||||
- Per-pod ephemeral names — just count total
|
||||
- Individual container IPs (no SVG variant now)
|
||||
- Per-volume mount details
|
||||
- Network IP ranges for individual services
|
||||
|
||||
4. **Endpoints that return 403 (skip these)**:
|
||||
- `GET /configs` — needs POST=1
|
||||
- `GET /secrets` — needs POST=1
|
||||
- `GET /info` — needs POST=1
|
||||
These exist but aren't readable through the proxy. Known secrets are documented in `mermaid.md` and don't need refreshing.
|
||||
|
||||
5. **Push** (NO force-push, just a normal commit on top):
|
||||
```bash
|
||||
cd /opt/data/work/diagram-cleanup # or wherever this worktree is
|
||||
# edit mermaid.md
|
||||
git add docs/architecture/mermaid.md
|
||||
git commit -m "docs: refresh prod map (snapshot YYYY-MM-DD)"
|
||||
git push origin docs/architecture-diagram-cleanup
|
||||
```
|
||||
Then open or update the PR against `admin/ansible:master`.
|
||||
|
||||
## What's intentionally NOT in the diagram
|
||||
|
||||
- Container IPs (was in old standalone-adaptive.svg, removed because too noisy)
|
||||
- Full list of 119 anonymous volumes (just count)
|
||||
- Per-mount details for each container
|
||||
- Internal stack service-to-service traffic (only ingress/egress/edge points shown)
|
||||
|
||||
## Known limitations / don't fix unless asked
|
||||
|
||||
- `endpoint.Ports` for `gitea_server` is empty but ingress still routes — legacy/race condition, leave alone
|
||||
- `nginx_nginx` service is Replicas=0 but overlay network `nginx_ingress` stays alive — orphan, leave alone
|
||||
- `act_runner` spawns ephemeral job containers (random names like `romantic_wozniak`) on every CI run — they accumulate as exited containers, that's normal
|
||||
- `brencher_tmp` volume gets re-populated by every `docker stack deploy` — that's how brencher works
|
||||
@@ -0,0 +1,253 @@
|
||||
# Architecture diagram — full prod map
|
||||
|
||||
> **Snapshot:** 2026-08-27 · sources: `docker_socket_proxy` + gitea admin API
|
||||
>
|
||||
> Single Mermaid view — high-level topology of all prod elements.
|
||||
> For per-node detail (IPs, container names), run the update commands in `README.md`
|
||||
> against a fresh snapshot.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
%% ==================== EXTERNAL ====================
|
||||
subgraph EXT[" "]
|
||||
direction TB
|
||||
Client["Client<br/>tailscale"]:::client
|
||||
Keenetic["Keenetic<br/>DNS · TLS"]:::external
|
||||
Tailscale["Tailscale<br/>subnet router"]:::external
|
||||
lan["lan<br/>192.168.1.0/24"]:::external
|
||||
|
||||
Client -->|https| Tailscale
|
||||
Tailscale --> Keenetic
|
||||
Keenetic -->|TLS term| lan
|
||||
end
|
||||
|
||||
%% ==================== INGRESS / EGRESS ====================
|
||||
subgraph IO[" "]
|
||||
direction TB
|
||||
ingress["ingress<br/>10.0.0.0/24"]:::egress
|
||||
proxy_egress["squid_egress<br/>hermes_proxy<br/>admin"]:::egress
|
||||
nginx_nets["nginx_ingress<br/>nginx_default<br/>(DEAD)"]:::egress
|
||||
|
||||
lan -.->|egress| ingress
|
||||
end
|
||||
|
||||
%% ==================== STACKS (deployed via brencher) ====================
|
||||
subgraph BRENCHER_DEPLOYED["via brencher (PROFILES env)"]
|
||||
direction TB
|
||||
b_gitea["gitea"]:::stack
|
||||
b_hermes["hermes"]:::stack
|
||||
b_hermes_infra["hermes_infra"]:::stack
|
||||
b_immich["immich"]:::stack
|
||||
b_gmail_mcp["gmail_mcp"]:::stack
|
||||
b_torrserv["torrserv_proxy"]:::stack
|
||||
b_brencher2["brencher2<br/>(deploy tool)"]:::deploy
|
||||
end
|
||||
|
||||
ingress --> b_gitea
|
||||
ingress --> b_hermes
|
||||
ingress --> b_hermes_infra
|
||||
ingress --> b_immich
|
||||
ingress --> b_gmail_mcp
|
||||
ingress --> b_torrserv
|
||||
|
||||
proxy_egress -.-> b_hermes_infra
|
||||
|
||||
b_brencher2 -.->|deploys| b_gitea
|
||||
b_brencher2 -.->|deploys| b_hermes
|
||||
b_brencher2 -.->|deploys| b_immich
|
||||
b_brencher2 -.->|deploys| b_hermes_infra
|
||||
b_brencher2 -.->|deploys| b_gmail_mcp
|
||||
b_brencher2 -.->|deploys| b_torrserv
|
||||
|
||||
%% ==================== STACKS (standalone, not via brencher) ====================
|
||||
subgraph STANDALONE["standalone (compose или ручной deploy)"]
|
||||
direction TB
|
||||
s_dsp["docker_socket_proxy"]:::stack
|
||||
s_lampa["lampa-jackett<br/>(compose)"]:::stack
|
||||
s_photosHelper["photosHelper<br/>(DEAD · Replicas: 0)"]:::dead
|
||||
s_nginx["nginx<br/>(DEAD · Replicas: 0)"]:::dead
|
||||
s_registry["registry<br/>(DEAD · Replicas: 0)"]:::dead
|
||||
end
|
||||
|
||||
ingress --> s_dsp
|
||||
ingress --> s_lampa
|
||||
|
||||
%% ==================== ZOMBIE STACKS (volumes only, no services) ====================
|
||||
subgraph ZOMBIE["zombie stacks (volumes orphaned)"]
|
||||
direction TB
|
||||
z_nextcloud["nextcloud<br/>(4 volumes)"]:::dead
|
||||
z_portainer["portainer<br/>(1 volume)"]:::dead
|
||||
z_swarmpit["swarmpit<br/>(1 volume)"]:::dead
|
||||
z_sstp["sstp · softether<br/>(4 volumes)"]:::dead
|
||||
z_test["test<br/>(1 volume)"]:::dead
|
||||
z_torr_cache["torr_cache<br/>(1 volume)"]:::dead
|
||||
z_beszel["beszel-poc<br/>(2 volumes)"]:::dead
|
||||
end
|
||||
|
||||
%% ==================== EPHEMERAL (gitea act_runner jobs) ====================
|
||||
subgraph EPHEMERAL["gitea act_runner ephemeral jobs"]
|
||||
direction TB
|
||||
eph_jobs["~48 exited<br/>ephemeral job pods<br/>(random names)"]:::ephemeral
|
||||
end
|
||||
|
||||
b_gitea -.->|spawns| eph_jobs
|
||||
|
||||
%% ==================== INTERNAL NETS (one per stack) ====================
|
||||
subgraph INTERNAL["internal nets (12)"]
|
||||
direction TB
|
||||
gitea_net["gitea_gitea"]:::internal
|
||||
hermes_net["hermes_hermes"]:::internal
|
||||
hi_net["hermes_infra_default"]:::internal
|
||||
immich_net["immich_default"]:::internal
|
||||
gmail_net["gmail_mcp_default"]:::internal
|
||||
torr_net["torrserv_proxy_default"]:::internal
|
||||
brencher2_nets["brencher2-net +<br/>brencher2_default"]:::internal
|
||||
dsp_net["dsp_default"]:::internal
|
||||
lampa_net["lampa_default"]:::internal
|
||||
|
||||
b_gitea --> gitea_net
|
||||
b_hermes --> hermes_net
|
||||
b_hermes_infra --> hi_net
|
||||
b_immich --> immich_net
|
||||
b_gmail_mcp --> gmail_net
|
||||
b_torrserv --> torr_net
|
||||
b_brencher2 --> brencher2_nets
|
||||
s_dsp --> dsp_net
|
||||
s_lampa --> lampa_net
|
||||
end
|
||||
|
||||
%% ==================== VOLUMES (153 total) ====================
|
||||
subgraph VOLS["volumes (153 total, 18 mounted · 135 orphaned)"]
|
||||
direction TB
|
||||
vol_active["active: gitea_data · immich_* ·<br/>hermes_*_data · brencher_tmp ·<br/>docker_socket_proxy_*_data ·<br/>torrserv_proxy_data · lampa_*"]:::volumes
|
||||
vol_orphaned["orphaned: 119 (no stack) ·<br/>nextcloud · portainer · sstp ·<br/>swarmpit · test · torr_cache ·<br/>beszel-poc · immich_postgres"]:::dead
|
||||
end
|
||||
|
||||
%% ==================== SECRETS / CONFIGS (403 from socket proxy) ====================
|
||||
subgraph SECRETS["secrets & configs (read-only proxy: 403)"]
|
||||
direction TB
|
||||
sec_known["known to exist in gitea app.ini:<br/>LFS_JWT_SECRET · JWT_SECRET ·<br/>SECRET_KEY · INTERNAL_TOKEN"]:::secret
|
||||
sec_hermes["hermes-agent runtime secrets<br/>(gmail creds · telegram token)"]:::secret
|
||||
end
|
||||
|
||||
%% ==================== STYLING ====================
|
||||
classDef client fill:#0f172a,stroke:#a78bfa,color:#ddd6fe
|
||||
classDef external fill:#1e293b,stroke:#94a3b8,color:#cbd5e1
|
||||
classDef egress fill:#052e16,stroke:#22c55e,color:#bbf7d0
|
||||
classDef stack fill:#0c4a6e,stroke:#22d3ee,color:#cffafe
|
||||
classDef dead fill:#1f1717,stroke:#fb7185,color:#fecaca,stroke-dasharray: 4 2
|
||||
classDef deploy fill:#422006,stroke:#fbbf24,color:#fde68a
|
||||
classDef ephemeral fill:#1e1b4b,stroke:#818cf8,color:#c7d2fe,stroke-dasharray: 2 2
|
||||
classDef internal fill:#1e1b4b,stroke:#818cf8,color:#c7d2fe
|
||||
classDef volumes fill:#0f172a,stroke:#475569,color:#cbd5e1
|
||||
classDef secret fill:#0f172a,stroke:#f59e0b,color:#fde68a,stroke-dasharray: 3 2
|
||||
```
|
||||
|
||||
## Полная картина прода
|
||||
|
||||
### 🟢 Running stacks (12)
|
||||
|
||||
**Через `brencher` (PROFILES env):**
|
||||
1. **gitea** — gitea_server + gitea_runner (act_runner для CI)
|
||||
2. **hermes** — hermes-agent + hermes-gateway
|
||||
3. **hermes_infra** — squid + sarg + hermes-ui + hermes-gateway + docker_socket_proxy + brencher-state-proxy
|
||||
4. **immich** — server + database + redis + machine-learning
|
||||
5. **gmail_mcp** — gmail-mcp-server + telegram-mcp-server
|
||||
6. **torrserv_proxy** — reverseproxy + fileserver
|
||||
7. **brencher2** — второй deploy tool (сам себя не деплоит)
|
||||
|
||||
**Standalone (не через brencher):**
|
||||
8. **docker_socket_proxy** — standalone, для hermes-gateway
|
||||
9. **lampa-jackett** — docker-compose, port 9117
|
||||
|
||||
**DEAD services (но стек определён):**
|
||||
10. **photosHelper** (Replicas: 0, 1 volume)
|
||||
11. **nginx** (Replicas: 0, 2 services, 5 volumes)
|
||||
12. **registry** (Replicas: 0, 1 volume)
|
||||
|
||||
### 🟡 Zombie stacks (volumes, но 0 running services)
|
||||
|
||||
Volumes остались после того как сервисы удалили. 4–5 volumes на стек:
|
||||
|
||||
- **nextcloud** (4 volumes · config + данные)
|
||||
- **sstp** (4 volumes · SoftEther VPN, конфиг + logs)
|
||||
- **portainer** (1 volume · swarm UI)
|
||||
- **swarmpit** (1 volume · swarm UI)
|
||||
- **test** (1 volume)
|
||||
- **torr_cache** (1 volume)
|
||||
- **beszel-poc** (2 volumes · monitoring POC)
|
||||
|
||||
### 🟣 Ephemeral: gitea act_runner job pods
|
||||
|
||||
Когда CI job запускается в gitea, `act_runner` создаёт **ephemeral контейнер** со случайным именем (`romantic_wozniak`, `compassionate_goodall` и т.п.). После завершения контейнер остаётся в `exited` state. На текущий момент ~48 таких exited контейнеров.
|
||||
|
||||
Реестр: image `ghcr.io/immich-app/runner-images:ubuntu-latest` (определён в `poc/gitea/config.yaml`).
|
||||
|
||||
### 🔴 Volumes (153 total)
|
||||
|
||||
- **18 mounted** в running контейнеры
|
||||
- **135 orphaned** (либо от zombie-стеков, либо anonymous)
|
||||
|
||||
Основные используемые:
|
||||
- `gitea_data` — критичные данные gitea (NFS на 192.168.1.96:/root/volumes/gitea)
|
||||
- `immich_*` — uploads, model-cache, postgres
|
||||
- `hermes_infra_*_data`, `hermes_hermes_*_data`
|
||||
- `brencher_tmp`, `brencher2_tmp` — git clones
|
||||
- `lampa_jackett_config`
|
||||
- `torrserv_proxy_data`
|
||||
|
||||
### 🔐 Secrets & configs
|
||||
|
||||
`/configs` и `/secrets` через socket-proxy **403** (требуется POST:1). Известные по другим источникам:
|
||||
|
||||
**В gitea `/data/gitea/conf/app.ini`:**
|
||||
- `LFS_JWT_SECRET`
|
||||
- `JWT_SECRET` (oauth2)
|
||||
- `SECRET_KEY`
|
||||
- `INTERNAL_TOKEN`
|
||||
|
||||
**В hermes-agent runtime:**
|
||||
- Gmail API creds (для gmail_mcp server)
|
||||
- Telegram bot token (для telegram_mcp server)
|
||||
|
||||
**Swarm configs** (из `poc/gitea/stack-compose.yml`):
|
||||
- `runner_config` — `config.yaml` для act_runner
|
||||
|
||||
### 🗑️ Dead-overlay сети (сеть есть, контейнеров нет)
|
||||
|
||||
- `nginx_ingress` (10.0.3.0/24) — для nginx-proxy, но nginx Replicas=0
|
||||
- `nginx_default` (10.0.4.0/24) — то же
|
||||
|
||||
Сети **живые** в swarm mesh, но не используются.
|
||||
|
||||
## Что НЕ показывает эта диаграмма
|
||||
|
||||
- Конкретные IP контейнеров (см. README.md — собираются через `docker_socket_proxy`)
|
||||
- Cross-stack сети в деталях (см. SVG)
|
||||
- Точные timestamps exited containers
|
||||
- Полный список 119 orphaned volumes без stack'а
|
||||
|
||||
## Как обновить
|
||||
|
||||
Запусти в этом окружении:
|
||||
|
||||
```bash
|
||||
# Stacks & services
|
||||
curl http://docker_socket_proxy:2375/services | jq -r '.[] | .Spec.Labels["com.docker.stack.namespace"]' | sort -u
|
||||
|
||||
# Volumes per stack
|
||||
curl http://docker_socket_proxy:2375/volumes | jq -r '.Volumes[] | "\(.Name) \(.Labels["com.docker.stack_namespace"] // "no-stack")"'
|
||||
|
||||
# Ephemeral containers (exited)
|
||||
curl http://docker_socket_proxy:2375/containers/json?all=1 | jq '[.[] | select(.State=="exited")] | length'
|
||||
|
||||
# BRENCHER profiles (env var)
|
||||
curl http://docker_socket_proxy:2375/containers/brencher_brencher-backend* 2>/dev/null
|
||||
```
|
||||
|
||||
## Связанные
|
||||
|
||||
- `README.md` — команды обновления этой диаграммы
|
||||
- `admin/ansible#5` — re-deploy hermes_squid
|
||||
- `admin/ansible#6` — pre-create socket_proxy overlay
|
||||
- `admin/ansible#7` — gitea stack deploy / app.ini fix
|
||||
Reference in New Issue
Block a user