feat: voice-gateway service (bridge browser to Hermes over WebSocket+PCM) #4

Closed
hermes wants to merge 1 commits from hermes/hermes-voice-gateway:feat/voice-gateway-mvp into main
Member

BLOCKED

Cannot merge until admin/ansible#7 (Gitea ROOT_URL fix) lands. The release workflow pushes images to Gitea Packages, which depends on the Gitea Container Registry endpoint emitting the public domain in /v2/token URLs. With internal IP 192.168.1.96 in the token URL (current bug), Docker push cannot complete.

Verified locally: run #121 of .gitea/workflows/release.yml fails on docker/login-action with dial tcp 192.168.1.96:80: connection refused.

Once #7 lands and app.ini is redeployed, re-trigger this workflow by re-pushing tag v0.1.0 (or bumping to v0.1.1).

Scope (unchanged)

Single PR combining the gateway service (PR B) and the browser client (PR C) from Org1/hermes-voice-gateway#3. Both halves are useless without each other.

Architecture

[Browser]  ⇄  [voice-gateway (rudolf-T)]  ⇄  [hermes-proxy]  ⇄  [Hermes via minimax API]
            WS+PCM 16 kHz in / 22050 Hz out     HTTP OpenAI-compat        text in / text out

Half-duplex: STT -> text -> LLM -> TTS -> audio. No barge-in, no duplex. ~2-5s turn latency.

Why STT/TTS bypass hermes-agent providers

Hermes-agent ships STT/TTS providers (OpenAI, Edge, Mistral, Piper, ...) but they all target file-based voice-bubble delivery (Telegram/Discord), not streaming PCM for a browser WebSocket. The gateway therefore uses faster-whisper + piper directly. The LLM step still goes through hermes-proxy because OpenAI-compatible chat-completions is exactly what the agent ships.

Files

  • gateway/server.py — aiohttp app: /ws WebSocket, /healthz, / static index.
  • gateway/pipeline.py — STT / LLM / TTS adapter classes.
  • gateway/config.py — env-driven config, no hidden defaults.
  • gateway/static/index.html + app.js + styles.css — browser mic capture (VAD-based silence flush) and WS playback.
  • Dockerfile — extends nousresearch/hermes-agent:v2026.8.3; adds faster-whisper, piper-tts, onnxruntime.
  • compose.yaml — Swarm stack, placement node.hostname==rudolf-T, joins hermes_proxy overlay to reach hermes-proxy by service name.
  • .gitea/workflows/release.yml — tag-triggered build & push to gitea.rudolf.keenetic.link/Org1/hermes-voice-gateway.

Related

## BLOCKED Cannot merge until [admin/ansible#7](https://gitea.rudolf.keenetic.link/admin/ansible/issues/7) (Gitea ROOT_URL fix) lands. The release workflow pushes images to Gitea Packages, which depends on the Gitea Container Registry endpoint emitting the public domain in `/v2/token` URLs. With internal IP `192.168.1.96` in the token URL (current bug), Docker push cannot complete. Verified locally: run #121 of `.gitea/workflows/release.yml` fails on `docker/login-action` with `dial tcp 192.168.1.96:80: connection refused`. Once #7 lands and `app.ini` is redeployed, re-trigger this workflow by re-pushing tag `v0.1.0` (or bumping to `v0.1.1`). ## Scope (unchanged) Single PR combining the gateway service (PR B) and the browser client (PR C) from Org1/hermes-voice-gateway#3. Both halves are useless without each other. ## Architecture ``` [Browser] ⇄ [voice-gateway (rudolf-T)] ⇄ [hermes-proxy] ⇄ [Hermes via minimax API] WS+PCM 16 kHz in / 22050 Hz out HTTP OpenAI-compat text in / text out ``` Half-duplex: STT -> text -> LLM -> TTS -> audio. No barge-in, no duplex. ~2-5s turn latency. ## Why STT/TTS bypass hermes-agent providers Hermes-agent ships STT/TTS providers (OpenAI, Edge, Mistral, Piper, ...) but they all target file-based voice-bubble delivery (Telegram/Discord), not streaming PCM for a browser WebSocket. The gateway therefore uses faster-whisper + piper directly. The LLM step still goes through hermes-proxy because OpenAI-compatible chat-completions is exactly what the agent ships. ## Files - `gateway/server.py` — aiohttp app: `/ws` WebSocket, `/healthz`, `/` static index. - `gateway/pipeline.py` — STT / LLM / TTS adapter classes. - `gateway/config.py` — env-driven config, no hidden defaults. - `gateway/static/index.html` + `app.js` + `styles.css` — browser mic capture (VAD-based silence flush) and WS playback. - `Dockerfile` — extends `nousresearch/hermes-agent:v2026.8.3`; adds faster-whisper, piper-tts, onnxruntime. - `compose.yaml` — Swarm stack, placement `node.hostname==rudolf-T`, joins `hermes_proxy` overlay to reach hermes-proxy by service name. - `.gitea/workflows/release.yml` — tag-triggered build & push to `gitea.rudolf.keenetic.link/Org1/hermes-voice-gateway`. ## Related - Org1/hermes-voice-gateway#1 (parent decision) - Org1/hermes-voice-gateway#3 (this implementation plan) - **admin/ansible#7** (BLOCKER — Gitea ROOT_URL fix) - hermes/ansible#4 (hermes-proxy service in hermes_infra stack)
hermes added 1 commit 2026-09-12 23:20:35 +02:00
Stack: rudolf-T worker, x86_64, 14.5 GiB RAM.
Replaces hermes-agent STT/TTS providers with direct faster-whisper +
piper (hermes-agent's TTS providers only target file-based voice-bubble
delivery, not streaming PCM for browser WebSocket).

Bridge: browser <-> voice-gateway <-> hermes-proxy <-> Hermes.

* gateway/server.py: aiohttp app with /ws, /healthz, / static
* gateway/pipeline.py: STT (faster-whisper base), LLM (POST
  /v1/chat/completions streaming), TTS (piper ru_RU-denis-medium)
* gateway/static/: browser mic capture + WS playback with VAD-based
  flush on silence
* Dockerfile on nousresearch/hermes-agent:v2026.8.3
* compose.yaml with placement node.hostname==rudolf-T

Wire protocol documented in server.py header and README. Half-duplex
by design; barge-in and server-VAD deferred until proven needed.
hermes changed title from feat: voice-gateway service (bridge browser to Hermes over WebSocket+PCM) to [blocked: admin/ansible#7] feat: voice-gateway service (bridge browser to Hermes over WebSocket+PCM) 2026-09-13 14:58:22 +02:00
hermes changed title from [blocked: admin/ansible#7] feat: voice-gateway service (bridge browser to Hermes over WebSocket+PCM) to feat: voice-gateway service (bridge browser to Hermes over WebSocket+PCM) 2026-09-13 16:06:11 +02:00
hermes closed this pull request 2026-09-13 16:54:13 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Reference: org1/hermes-voice-gateway#4