forked from org1/hermes-voice-gateway
feat: voice-gateway service (PR B + PR C in single PR)
Stack: rudolf-T worker, x86_64, 14.5 GiB RAM. Replaces hermes-agent STT/TTS providers with direct faster-whisper + piper (hermes-agent's TTS providers only target file-based voice-bubble delivery, not streaming PCM for browser WebSocket). Bridge: browser <-> voice-gateway <-> hermes-proxy <-> Hermes. * gateway/server.py: aiohttp app with /ws, /healthz, / static * gateway/pipeline.py: STT (faster-whisper base), LLM (POST /v1/chat/completions streaming), TTS (piper ru_RU-denis-medium) * gateway/static/: browser mic capture + WS playback with VAD-based flush on silence * Dockerfile on nousresearch/hermes-agent:v2026.8.3 * compose.yaml with placement node.hostname==rudolf-T Wire protocol documented in server.py header and README. Half-duplex by design; barge-in and server-VAD deferred until proven needed.
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
.env
|
||||
.venv/
|
||||
node_modules/
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
FROM nousresearch/hermes-agent:v2026.8.3
|
||||
|
||||
RUN pip install --no-cache-dir \
|
||||
"websockets>=12,<14" \
|
||||
"aiohttp>=3.9" \
|
||||
"faster-whisper>=1.0" \
|
||||
"piper-tts>=1.2" \
|
||||
"onnxruntime>=1.17"
|
||||
|
||||
WORKDIR /app
|
||||
COPY gateway/ /app/
|
||||
|
||||
EXPOSE 8765
|
||||
CMD ["python", "-m", "gateway.server"]
|
||||
@@ -0,0 +1,69 @@
|
||||
# hermes-voice-gateway
|
||||
|
||||
Local voice realtime bridge: browser ↔ voice-gateway (WebSocket+PCM) ↔ hermes-proxy (OpenAI-compatible) ↔ Hermes (minimax API).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
[Browser] [voice-gateway] [hermes-proxy] [Hermes]
|
||||
mic/speaker ⇄ WS+PCM on :8765 ⇄ POST /v1/chat ⇄ minimax API
|
||||
│
|
||||
├─ STT: faster-whisper base (VAD-filtered)
|
||||
└─ TTS: piper ru_RU-denis-medium
|
||||
```
|
||||
|
||||
Half-duplex by design: client sends PCM chunks, server flushes STT on silence, calls LLM with streaming, splits reply into sentences, sends each sentence to TTS, streams PCM back.
|
||||
|
||||
## Stack layout
|
||||
|
||||
- `gateway/server.py` — aiohttp WS endpoint + HTTP static
|
||||
- `gateway/pipeline.py` — STT / LLM / TTS adapter classes
|
||||
- `gateway/config.py` — env-driven config
|
||||
- `gateway/static/{index.html,app.js,styles.css}` — browser client
|
||||
- `Dockerfile` — image build on top of `nousresearch/hermes-agent:v2026.8.3`
|
||||
- `compose.yaml` — Swarm stack with placement on `rudolf-T`
|
||||
|
||||
## Required env
|
||||
|
||||
| Var | Default | Notes |
|
||||
|---|---|---|
|
||||
| `HERMES_PROXY_URL` | `http://hermes-proxy:8645` | OpenAI-compatible upstream |
|
||||
| `HERMES_PROXY_TOKEN` | empty | Bearer if hermes-proxy requires one |
|
||||
| `PIPER_VOICE_PATH` | `/opt/piper-voices/ru_RU-denis-medium.onnx` | bundled in `voice_gateway_models` volume |
|
||||
| `LLM_MODEL` | `MiniMax-M3` | passed verbatim to /v1/chat/completions |
|
||||
| `SAMPLE_RATE_IN` | `16000` | PCM in from browser |
|
||||
| `SAMPLE_RATE_OUT` | `22050` | PCM out to browser |
|
||||
| `MAX_SESSION_SECONDS` | `300` | hard timeout per WS session |
|
||||
| `MAX_SILENCE_MS` | `1500` | browser-side VAD trigger |
|
||||
|
||||
## Wire protocol
|
||||
|
||||
Client → server (WS):
|
||||
- binary frame: PCM Int16 mono at `SAMPLE_RATE_IN`
|
||||
- text JSON `{"type": "flush"}` — force STT
|
||||
- text JSON `{"type": "reset"}` — clear conversation history
|
||||
- text JSON `{"type": "ping"}` — keep-alive
|
||||
|
||||
Server → client (WS):
|
||||
- binary frame: PCM Int16 mono at `SAMPLE_RATE_OUT`
|
||||
- text JSON `{"type": "ready"}` — handshake complete
|
||||
- text JSON `{"type": "transcript", "text": "..."}`
|
||||
- text JSON `{"type": "reply", "text": "..."}` — full assistant text after TTS finishes
|
||||
- text JSON `{"type": "error", "message": "..."}`
|
||||
|
||||
## Running locally
|
||||
|
||||
```
|
||||
docker compose build
|
||||
docker compose up
|
||||
open http://localhost:8765/
|
||||
```
|
||||
|
||||
The voice-gateway expects `hermes-proxy` reachable at `http://hermes-proxy:8645` on the same `hermes_proxy` overlay. In local dev, override `HERMES_PROXY_URL=http://host.docker.internal:8645`.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `nousresearch/hermes-agent:v2026.8.3` — base image, brings hermes-agent CLI (not strictly required by gateway code, but matches the rest of hermes_infra).
|
||||
- `faster-whisper` — STT, model `base` int8 quantized, downloaded on first use.
|
||||
- `piper-tts` + `onnxruntime` — TTS, voice model must be mounted at `PIPER_VOICE_PATH`.
|
||||
- `aiohttp` — HTTP/WS server.
|
||||
@@ -0,0 +1,63 @@
|
||||
name: hermes-voice-gateway
|
||||
|
||||
services:
|
||||
voice-gateway:
|
||||
image: registry.rudolf.keenetic.link/hermes-voice-gateway:${version}
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
hostname: voice-gateway
|
||||
user: "0:0"
|
||||
networks:
|
||||
- hermes_proxy
|
||||
ports:
|
||||
- target: 8765
|
||||
published: 8765
|
||||
protocol: tcp
|
||||
mode: host
|
||||
volumes:
|
||||
- voice_gateway_models:/opt/piper-voices:ro
|
||||
environment:
|
||||
VOICE_GATEWAY_HOST: "0.0.0.0"
|
||||
VOICE_GATEWAY_PORT: "8765"
|
||||
HERMES_PROXY_URL: "http://hermes-proxy:8645"
|
||||
HERMES_PROXY_TOKEN: "${HERMES_PROXY_TOKEN:-}"
|
||||
SAMPLE_RATE_IN: "16000"
|
||||
SAMPLE_RATE_OUT: "22050"
|
||||
LLM_MODEL: "MiniMax-M3"
|
||||
PIPER_VOICE_PATH: "/opt/piper-voices/ru_RU-denis-medium.onnx"
|
||||
MAX_SESSION_SECONDS: "300"
|
||||
MAX_SILENCE_MS: "1500"
|
||||
HTTP_PROXY: "http://squid:3128"
|
||||
HTTPS_PROXY: "http://squid:3128"
|
||||
NO_PROXY: "localhost,127.0.0.1,.local,squid,hermes-gateway,hermes-agent,hermes-proxy,brencher-state-proxy,docker_socket_proxy,10.0.0.0/8"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "python -c \"import urllib.request,sys; urllib.request.urlopen('http://127.0.0.1:8765/healthz', timeout=3).read()\" || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
deploy:
|
||||
mode: replicated
|
||||
replicas: 1
|
||||
placement:
|
||||
constraints:
|
||||
- node.hostname == rudolf-T
|
||||
resources:
|
||||
limits:
|
||||
memory: 4G
|
||||
pids: 1024
|
||||
reservations:
|
||||
memory: 1G
|
||||
restart_policy:
|
||||
condition: on-failure
|
||||
delay: 10s
|
||||
max_attempts: 3
|
||||
|
||||
networks:
|
||||
hermes_proxy:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
voice_gateway_models:
|
||||
driver: local
|
||||
@@ -0,0 +1,74 @@
|
||||
"""voice-gateway runtime configuration.
|
||||
|
||||
All values come from environment variables; no defaults that would change
|
||||
behavior silently. The compose file in this repo owns the canonical env,
|
||||
but local dev can override via a .env.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
raw = os.environ.get(name)
|
||||
if raw is None or raw == "":
|
||||
return default
|
||||
return int(raw)
|
||||
|
||||
|
||||
def _env_float(name: str, default: float) -> float:
|
||||
raw = os.environ.get(name)
|
||||
if raw is None or raw == "":
|
||||
return default
|
||||
return float(raw)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Config:
|
||||
ws_host: str
|
||||
ws_port: int
|
||||
|
||||
hermes_proxy_url: str
|
||||
hermes_proxy_token: str
|
||||
|
||||
sample_rate_in: int
|
||||
sample_rate_out: int
|
||||
|
||||
llm_model: str
|
||||
llm_system_prompt: str
|
||||
|
||||
stt_provider: str
|
||||
tts_provider: str
|
||||
|
||||
max_session_seconds: float
|
||||
max_silence_ms: int
|
||||
|
||||
|
||||
def load_config() -> Config:
|
||||
return Config(
|
||||
ws_host=os.environ.get("VOICE_GATEWAY_HOST", "0.0.0.0"),
|
||||
ws_port=_env_int("VOICE_GATEWAY_PORT", 8765),
|
||||
|
||||
hermes_proxy_url=os.environ.get(
|
||||
"HERMES_PROXY_URL", "http://hermes-proxy:8645"
|
||||
),
|
||||
hermes_proxy_token=os.environ.get("HERMES_PROXY_TOKEN", ""),
|
||||
|
||||
sample_rate_in=_env_int("SAMPLE_RATE_IN", 16000),
|
||||
sample_rate_out=_env_int("SAMPLE_RATE_OUT", 22050),
|
||||
|
||||
llm_model=os.environ.get("LLM_MODEL", "MiniMax-M3"),
|
||||
llm_system_prompt=os.environ.get(
|
||||
"LLM_SYSTEM_PROMPT",
|
||||
"You are a concise voice assistant. Reply in the user's language. "
|
||||
"Keep answers under 60 words unless asked for detail.",
|
||||
),
|
||||
|
||||
stt_provider=os.environ.get("STT_PROVIDER", "faster_whisper"),
|
||||
tts_provider=os.environ.get("TTS_PROVIDER", "piper"),
|
||||
|
||||
max_session_seconds=_env_float("MAX_SESSION_SECONDS", 300.0),
|
||||
max_silence_ms=_env_int("MAX_SILENCE_MS", 1500),
|
||||
)
|
||||
@@ -0,0 +1,153 @@
|
||||
"""Audio pipeline: PCM in -> STT -> LLM -> TTS -> PCM out.
|
||||
|
||||
STT and TTS are NOT routed through hermes-agent providers because the
|
||||
agent's TTS providers all target file-based voice-bubble delivery
|
||||
(Telegram/Discord), not streaming PCM for a browser WebSocket. Instead,
|
||||
gateway owns faster-whisper and piper directly as subprocess-backed
|
||||
helpers. The LLM step does use the OpenAI-compatible hermes-proxy.
|
||||
|
||||
Wire format:
|
||||
- Input: 16 kHz mono Int16 PCM, chunked into ~250ms frames by the client.
|
||||
- Output: 22050 Hz mono Int16 PCM, one chunk per TTS sentence.
|
||||
- LLM: POST /v1/chat/completions on hermes-proxy (OpenAI-compatible).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import logging
|
||||
import wave
|
||||
from typing import AsyncIterator, Optional
|
||||
|
||||
import aiohttp
|
||||
|
||||
from .config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class STT:
|
||||
def __init__(self, sample_rate: int) -> None:
|
||||
self.sample_rate = sample_rate
|
||||
self._buffer = bytearray()
|
||||
self._model = None
|
||||
|
||||
def _ensure_model(self) -> None:
|
||||
if self._model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
self._model = WhisperModel("base", compute_type="int8")
|
||||
return self._model
|
||||
|
||||
def feed_pcm(self, pcm_bytes: bytes) -> None:
|
||||
self._buffer.extend(pcm_bytes)
|
||||
|
||||
async def transcribe(self) -> str:
|
||||
if not self._buffer:
|
||||
return ""
|
||||
buf = bytes(self._buffer)
|
||||
self._buffer.clear()
|
||||
model = await asyncio.to_thread(self._ensure_model)
|
||||
|
||||
def _run() -> str:
|
||||
wav_buf = io.BytesIO()
|
||||
with wave.open(wav_buf, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2)
|
||||
wf.setframerate(self.sample_rate)
|
||||
wf.writeframes(buf)
|
||||
wav_buf.seek(0)
|
||||
segments, _ = model.transcribe(wav_buf, language=None, vad_filter=True)
|
||||
return " ".join(seg.text.strip() for seg in segments).strip()
|
||||
|
||||
text = await asyncio.to_thread(_run)
|
||||
return text
|
||||
|
||||
|
||||
class TTS:
|
||||
def __init__(self, sample_rate: int) -> None:
|
||||
self.sample_rate = sample_rate
|
||||
self._piper = None
|
||||
|
||||
def _ensure_voice(self):
|
||||
if self._piper is None:
|
||||
from piper import PiperVoice
|
||||
import os
|
||||
model_path = os.environ.get(
|
||||
"PIPER_VOICE_PATH", "/opt/piper-voices/ru_RU-denis-medium.onnx"
|
||||
)
|
||||
self._piper = PiperVoice.load(model_path, config_path=None)
|
||||
return self._piper
|
||||
|
||||
async def synth_stream(self, text: str) -> AsyncIterator[bytes]:
|
||||
if not text:
|
||||
return
|
||||
voice = await asyncio.to_thread(self._ensure_voice)
|
||||
|
||||
def _synth() -> bytes:
|
||||
buf = io.BytesIO()
|
||||
with wave.open(buf, "wb") as wf:
|
||||
voice.synthesize(text, wf)
|
||||
return buf.getvalue()
|
||||
|
||||
wav = await asyncio.to_thread(_synth)
|
||||
with wave.open(io.BytesIO(wav), "rb") as wf:
|
||||
frames = wf.readframes(wf.getnframes())
|
||||
chunk_size = self.sample_rate * 2 // 4
|
||||
for i in range(0, len(frames), chunk_size):
|
||||
yield frames[i:i + chunk_size]
|
||||
|
||||
|
||||
class LLM:
|
||||
def __init__(self, config: Config) -> None:
|
||||
self.config = config
|
||||
self._session: Optional[aiohttp.ClientSession] = None
|
||||
|
||||
async def _session_get(self) -> aiohttp.ClientSession:
|
||||
if self._session is None or self._session.closed:
|
||||
headers = {}
|
||||
if self.config.hermes_proxy_token:
|
||||
headers["Authorization"] = f"Bearer {self.config.hermes_proxy_token}"
|
||||
self._session = aiohttp.ClientSession(
|
||||
base_url=self.config.hermes_proxy_url,
|
||||
headers=headers,
|
||||
timeout=aiohttp.ClientTimeout(total=60),
|
||||
)
|
||||
return self._session
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._session and not self._session.closed:
|
||||
await self._session.close()
|
||||
|
||||
async def complete(
|
||||
self, transcript: str, history: list[dict]
|
||||
) -> AsyncIterator[str]:
|
||||
messages = list(history)
|
||||
messages.append({"role": "user", "content": transcript})
|
||||
session = await self._session_get()
|
||||
payload = {
|
||||
"model": self.config.llm_model,
|
||||
"messages": messages,
|
||||
"stream": True,
|
||||
"max_tokens": 200,
|
||||
"temperature": 0.7,
|
||||
}
|
||||
async with session.post("/v1/chat/completions", json=payload) as resp:
|
||||
resp.raise_for_status()
|
||||
async for raw in resp.content:
|
||||
if not raw:
|
||||
continue
|
||||
line = raw.decode("utf-8", errors="ignore").strip()
|
||||
if not line.startswith("data: "):
|
||||
continue
|
||||
data = line[6:]
|
||||
if data == "[DONE]":
|
||||
return
|
||||
import json
|
||||
try:
|
||||
obj = json.loads(data)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
delta = obj.get("choices", [{}])[0].get("delta", {}).get("content")
|
||||
if delta:
|
||||
yield delta
|
||||
@@ -0,0 +1,165 @@
|
||||
"""aiohttp server: WebSocket audio bridge between browser and hermes-proxy.
|
||||
|
||||
Wire protocol (WS frames):
|
||||
- Client -> server binary: PCM Int16 mono frames at cfg.sample_rate_in.
|
||||
- Client -> server text JSON: {"type": "flush"} to force STT on partial audio.
|
||||
- Client -> server text JSON: {"type": "reset"} clears conversation history.
|
||||
- Server -> client binary: PCM Int16 mono frames at cfg.sample_rate_out.
|
||||
- Server -> client text JSON: {"type": "ready"}.
|
||||
- Server -> client text JSON: {"type": "transcript", "text": "..."}.
|
||||
- Server -> client text JSON: {"type": "reply", "text": "..."}.
|
||||
- Server -> client text JSON: {"type": "error", "message": "..."}.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from aiohttp import WSMsgType, web
|
||||
|
||||
from .config import Config, load_config
|
||||
from .pipeline import LLM, STT, TTS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SENTENCE_SEPARATORS = (". ", "! ", "? ", "\n")
|
||||
|
||||
|
||||
async def _speak(text: str, tts: TTS, ws: web.WebSocketResponse) -> None:
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return
|
||||
async for chunk in tts.synth_stream(text):
|
||||
await ws.send_bytes(chunk)
|
||||
|
||||
|
||||
async def _flush_sentence(
|
||||
buffer: list[str], tts: TTS, ws: web.WebSocketResponse
|
||||
) -> None:
|
||||
text = "".join(buffer).strip()
|
||||
buffer.clear()
|
||||
await _speak(text, tts, ws)
|
||||
|
||||
|
||||
async def _run_turn(
|
||||
ws: web.WebSocketResponse,
|
||||
cfg: Config,
|
||||
stt: STT,
|
||||
llm: LLM,
|
||||
tts: TTS,
|
||||
history: list[dict],
|
||||
) -> None:
|
||||
transcript = await stt.transcribe()
|
||||
if not transcript:
|
||||
await ws.send_json({"type": "transcript", "text": ""})
|
||||
return
|
||||
|
||||
await ws.send_json({"type": "transcript", "text": transcript})
|
||||
history.append({"role": "user", "content": transcript})
|
||||
|
||||
sentence_buf: list[str] = []
|
||||
full_reply: list[str] = []
|
||||
tts_started = time.monotonic()
|
||||
|
||||
async for delta in llm.complete(transcript, history[:-1]):
|
||||
full_reply.append(delta)
|
||||
sentence_buf.append(delta)
|
||||
joined = "".join(sentence_buf)
|
||||
for sep in _SENTENCE_SEPARATORS:
|
||||
while sep in joined:
|
||||
cut = joined.find(sep) + len(sep)
|
||||
head, joined = joined[:cut], joined[cut:]
|
||||
await _flush_sentence([head], tts, ws)
|
||||
sentence_buf.clear()
|
||||
sentence_buf.append(joined)
|
||||
|
||||
await _flush_sentence(sentence_buf, tts, ws)
|
||||
|
||||
reply_text = "".join(full_reply).strip()
|
||||
history.append({"role": "assistant", "content": reply_text})
|
||||
await ws.send_json({"type": "reply", "text": reply_text})
|
||||
logger.info(
|
||||
"turn done: stt_chars=%d reply_chars=%d tts_ms=%.0f",
|
||||
len(transcript), len(reply_text), (time.monotonic() - tts_started) * 1000,
|
||||
)
|
||||
|
||||
|
||||
async def _session(ws: web.WebSocketResponse, cfg: Config) -> None:
|
||||
await ws.send_json({"type": "ready"})
|
||||
stt = STT(cfg.sample_rate_in)
|
||||
tts = TTS(cfg.sample_rate_out)
|
||||
llm = LLM(cfg)
|
||||
history: list[dict] = [
|
||||
{"role": "system", "content": cfg.llm_system_prompt}
|
||||
]
|
||||
deadline = time.monotonic() + cfg.max_session_seconds
|
||||
|
||||
try:
|
||||
async for msg in ws:
|
||||
if msg.type == WSMsgType.BINARY:
|
||||
stt.feed_pcm(msg.data)
|
||||
elif msg.type == WSMsgType.TEXT:
|
||||
try:
|
||||
cmd = json.loads(msg.data)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
ctype = cmd.get("type")
|
||||
if ctype == "flush":
|
||||
if time.monotonic() > deadline:
|
||||
await ws.send_json({"type": "error", "message": "session_timeout"})
|
||||
return
|
||||
await _run_turn(ws, cfg, stt, llm, tts, history)
|
||||
elif ctype == "reset":
|
||||
history.clear()
|
||||
history.append({"role": "system", "content": cfg.llm_system_prompt})
|
||||
elif ctype == "ping":
|
||||
await ws.send_json({"type": "pong"})
|
||||
elif msg.type == WSMsgType.ERROR:
|
||||
logger.warning("ws error: %s", ws.exception())
|
||||
break
|
||||
finally:
|
||||
await llm.close()
|
||||
|
||||
|
||||
async def ws_handler(request: web.Request) -> web.WebSocketResponse:
|
||||
cfg: Config = request.app["cfg"]
|
||||
ws = web.WebSocketResponse(heartbeat=30, max_msg_size=2 ** 20)
|
||||
await ws.prepare(request)
|
||||
await _session(ws, cfg)
|
||||
return ws
|
||||
|
||||
|
||||
async def healthz(request: web.Request) -> web.Response:
|
||||
return web.json_response({"status": "ok"})
|
||||
|
||||
|
||||
async def index(request: web.Request) -> web.Response:
|
||||
return web.FileResponse("gateway/static/index.html")
|
||||
|
||||
|
||||
def make_app() -> web.Application:
|
||||
app = web.Application(client_max_size=64 * 1024)
|
||||
app["cfg"] = load_config()
|
||||
app.router.add_get("/ws", ws_handler)
|
||||
app.router.add_get("/healthz", healthz)
|
||||
app.router.add_get("/", index)
|
||||
app.router.add_static("/static/", path="gateway/static", show_index=False)
|
||||
return app
|
||||
|
||||
|
||||
def main() -> None:
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||
)
|
||||
cfg = load_config()
|
||||
app = make_app()
|
||||
web.run_app(app, host=cfg.ws_host, port=cfg.ws_port, access_log=None)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,231 @@
|
||||
"use strict";
|
||||
|
||||
const SAMPLE_RATE_IN = 16000;
|
||||
const SAMPLE_RATE_OUT = 22050;
|
||||
const FRAME_MS = 100;
|
||||
const SILENCE_DB = -45;
|
||||
const SILENCE_HOLD_MS = 600;
|
||||
|
||||
const el = (id) => document.getElementById(id);
|
||||
const status = el("status");
|
||||
const log = el("log");
|
||||
const levelEl = el("level");
|
||||
|
||||
const connectBtn = el("connect");
|
||||
const disconnectBtn = el("disconnect");
|
||||
const holdBtn = el("hold");
|
||||
const resetBtn = el("reset");
|
||||
|
||||
let ws = null;
|
||||
let audioCtx = null;
|
||||
let micStream = null;
|
||||
let micNode = null;
|
||||
let playbackCtx = null;
|
||||
let nextPlayTime = 0;
|
||||
let recording = false;
|
||||
let silenceStart = 0;
|
||||
let lastSpeech = 0;
|
||||
let rafId = 0;
|
||||
let pcmBuf = [];
|
||||
|
||||
function setStatus(text, klass) {
|
||||
status.textContent = text;
|
||||
status.className = "status" + (klass ? " " + klass : "");
|
||||
}
|
||||
|
||||
function appendLog(role, text) {
|
||||
const div = document.createElement("div");
|
||||
div.className = role;
|
||||
div.textContent = (role === "user" ? "you: " : role === "assistant" ? "hermes: " : "* ") + text;
|
||||
log.appendChild(div);
|
||||
log.scrollTop = log.scrollHeight;
|
||||
}
|
||||
|
||||
function dbfs(rms) {
|
||||
if (rms <= 0) return -100;
|
||||
return 20 * Math.log10(rms);
|
||||
}
|
||||
|
||||
function frameSamples() {
|
||||
return SAMPLE_RATE_IN * FRAME_MS / 1000;
|
||||
}
|
||||
|
||||
function flushPCM() {
|
||||
if (ws && ws.readyState === WebSocket.OPEN && pcmBuf.length) {
|
||||
const total = pcmBuf.reduce((n, a) => n + a.length, 0);
|
||||
const merged = new Int16Array(total);
|
||||
let off = 0;
|
||||
for (const arr of pcmBuf) { merged.set(arr, off); off += arr.length; }
|
||||
ws.send(merged.buffer);
|
||||
pcmBuf = [];
|
||||
}
|
||||
}
|
||||
|
||||
function handleAudio(samples) {
|
||||
let sum = 0;
|
||||
for (let i = 0; i < samples.length; i++) sum += samples[i] * samples[i];
|
||||
const rms = Math.sqrt(sum / samples.length);
|
||||
const db = dbfs(rms);
|
||||
levelEl.value = Math.max(0, Math.min(100, 100 + db));
|
||||
|
||||
if (db > SILENCE_DB) {
|
||||
lastSpeech = performance.now();
|
||||
silenceStart = 0;
|
||||
pcmBuf.push(samples);
|
||||
if (pcmBuf.reduce((n, a) => n + a.length, 0) >= SAMPLE_RATE_IN * 1.0) {
|
||||
flushPCM();
|
||||
}
|
||||
} else if (!silenceStart) {
|
||||
silenceStart = performance.now();
|
||||
} else if (performance.now() - silenceStart >= SILENCE_HOLD_MS) {
|
||||
flushPCM();
|
||||
if (ws && ws.readyState === WebSocket.OPEN && lastSpeech && performance.now() - lastSpeech >= SILENCE_HOLD_MS) {
|
||||
ws.send(JSON.stringify({ type: "flush" }));
|
||||
}
|
||||
silenceStart = 0;
|
||||
lastSpeech = 0;
|
||||
}
|
||||
}
|
||||
|
||||
async function startMic() {
|
||||
audioCtx = new AudioContext({ sampleRate: SAMPLE_RATE_IN });
|
||||
micStream = await navigator.mediaDevices.getUserMedia({
|
||||
audio: {
|
||||
channelCount: 1,
|
||||
sampleRate: SAMPLE_RATE_IN,
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
autoGainControl: true,
|
||||
},
|
||||
video: false,
|
||||
});
|
||||
const src = audioCtx.createMediaStreamSource(micStream);
|
||||
micNode = audioCtx.createScriptProcessor(2048, 1, 1);
|
||||
src.connect(micNode);
|
||||
micNode.connect(audioCtx.destination);
|
||||
micNode.onaudioprocess = (e) => {
|
||||
if (!recording) return;
|
||||
const ch = e.inputBuffer.getChannelData(0);
|
||||
const pcm = new Int16Array(ch.length);
|
||||
for (let i = 0; i < ch.length; i++) {
|
||||
const s = Math.max(-1, Math.min(1, ch[i]));
|
||||
pcm[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
|
||||
}
|
||||
handleAudio(pcm);
|
||||
};
|
||||
}
|
||||
|
||||
function ensurePlayback() {
|
||||
if (!playbackCtx) {
|
||||
playbackCtx = new AudioContext({ sampleRate: SAMPLE_RATE_OUT });
|
||||
}
|
||||
if (playbackCtx.state === "suspended") {
|
||||
playbackCtx.resume();
|
||||
}
|
||||
return playbackCtx;
|
||||
}
|
||||
|
||||
function playPcm(pcm) {
|
||||
const ctx = ensurePlayback();
|
||||
const buf = ctx.createBuffer(1, pcm.length, SAMPLE_RATE_OUT);
|
||||
const ch = buf.getChannelData(0);
|
||||
for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / (pcm[i] < 0 ? 0x8000 : 0x7fff);
|
||||
const src = ctx.createBufferSource();
|
||||
src.buffer = buf;
|
||||
src.connect(ctx.destination);
|
||||
const startAt = Math.max(ctx.currentTime, nextPlayTime);
|
||||
src.start(startAt);
|
||||
nextPlayTime = startAt + buf.duration;
|
||||
}
|
||||
|
||||
async function connect() {
|
||||
try {
|
||||
const proto = location.protocol === "https:" ? "wss:" : "ws:";
|
||||
ws = new WebSocket(`${proto}//${location.host}/ws`);
|
||||
ws.binaryType = "arraybuffer";
|
||||
ws.onopen = () => {
|
||||
setStatus("connected", "ok");
|
||||
connectBtn.disabled = true;
|
||||
disconnectBtn.disabled = false;
|
||||
holdBtn.disabled = false;
|
||||
resetBtn.disabled = false;
|
||||
appendLog("system", "websocket open");
|
||||
};
|
||||
ws.onmessage = (ev) => {
|
||||
if (ev.data instanceof ArrayBuffer) {
|
||||
playPcm(new Int16Array(ev.data));
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const msg = JSON.parse(ev.data);
|
||||
if (msg.type === "ready") appendLog("system", "ready");
|
||||
else if (msg.type === "transcript") appendLog("user", msg.text || "(empty)");
|
||||
else if (msg.type === "reply") appendLog("assistant", msg.text);
|
||||
else if (msg.type === "error") {
|
||||
appendLog("system", "error: " + msg.message);
|
||||
setStatus("error: " + msg.message, "err");
|
||||
}
|
||||
} catch (_) {}
|
||||
};
|
||||
ws.onclose = () => {
|
||||
setStatus("disconnected");
|
||||
connectBtn.disabled = false;
|
||||
disconnectBtn.disabled = true;
|
||||
holdBtn.disabled = true;
|
||||
resetBtn.disabled = true;
|
||||
recording = false;
|
||||
};
|
||||
ws.onerror = () => setStatus("ws error", "err");
|
||||
await startMic();
|
||||
} catch (err) {
|
||||
setStatus("connect failed: " + err.message, "err");
|
||||
}
|
||||
}
|
||||
|
||||
function disconnect() {
|
||||
if (ws) ws.close();
|
||||
if (micStream) micStream.getTracks().forEach((t) => t.stop());
|
||||
if (audioCtx) audioCtx.close();
|
||||
if (playbackCtx) playbackCtx.close();
|
||||
audioCtx = playbackCtx = micStream = micNode = null;
|
||||
recording = false;
|
||||
}
|
||||
|
||||
function startTalking() {
|
||||
if (!ws || ws.readyState !== WebSocket.OPEN) return;
|
||||
recording = true;
|
||||
silenceStart = 0;
|
||||
lastSpeech = 0;
|
||||
pcmBuf = [];
|
||||
holdBtn.textContent = "Release to send";
|
||||
holdBtn.classList.add("danger");
|
||||
}
|
||||
|
||||
function stopTalking() {
|
||||
if (!recording) return;
|
||||
recording = false;
|
||||
flushPCM();
|
||||
if (ws && ws.readyState === WebSocket.OPEN) {
|
||||
ws.send(JSON.stringify({ type: "flush" }));
|
||||
}
|
||||
holdBtn.textContent = "Hold to talk";
|
||||
holdBtn.classList.remove("danger");
|
||||
}
|
||||
|
||||
function reset() {
|
||||
if (ws && ws.readyState === WebSocket.OPEN) {
|
||||
ws.send(JSON.stringify({ type: "reset" }));
|
||||
log.innerHTML = "";
|
||||
nextPlayTime = 0;
|
||||
appendLog("system", "conversation reset");
|
||||
}
|
||||
}
|
||||
|
||||
connectBtn.addEventListener("click", connect);
|
||||
disconnectBtn.addEventListener("click", disconnect);
|
||||
resetBtn.addEventListener("click", reset);
|
||||
holdBtn.addEventListener("mousedown", startTalking);
|
||||
holdBtn.addEventListener("mouseup", stopTalking);
|
||||
holdBtn.addEventListener("mouseleave", stopTalking);
|
||||
holdBtn.addEventListener("touchstart", (e) => { e.preventDefault(); startTalking(); });
|
||||
holdBtn.addEventListener("touchend", (e) => { e.preventDefault(); stopTalking(); });
|
||||
@@ -0,0 +1,33 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="ru">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Hermes Voice</title>
|
||||
<link rel="stylesheet" href="/static/styles.css">
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<header>
|
||||
<h1>Hermes Voice</h1>
|
||||
<p class="status" id="status">disconnected</p>
|
||||
</header>
|
||||
|
||||
<div class="controls">
|
||||
<button id="connect" type="button">Connect</button>
|
||||
<button id="disconnect" type="button" disabled>Disconnect</button>
|
||||
<button id="hold" type="button" disabled>Hold to talk</button>
|
||||
<button id="reset" type="button" disabled>Reset</button>
|
||||
</div>
|
||||
|
||||
<section class="log" id="log" aria-live="polite"></section>
|
||||
|
||||
<section class="meter">
|
||||
<label>Mic level</label>
|
||||
<progress id="level" value="0" max="100"></progress>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<script src="/static/app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,90 @@
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--bg: #0e1116;
|
||||
--fg: #e6edf3;
|
||||
--muted: #8b949e;
|
||||
--accent: #58a6ff;
|
||||
--warn: #f85149;
|
||||
--ok: #3fb950;
|
||||
font-family: system-ui, sans-serif;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
background: var(--bg);
|
||||
color: var(--fg);
|
||||
min-height: 100vh;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
}
|
||||
|
||||
main {
|
||||
width: min(640px, 92vw);
|
||||
padding: 1.5rem;
|
||||
}
|
||||
|
||||
header h1 {
|
||||
margin: 0 0 .25rem 0;
|
||||
font-size: 1.4rem;
|
||||
}
|
||||
|
||||
.status {
|
||||
margin: 0;
|
||||
font-size: .85rem;
|
||||
color: var(--muted);
|
||||
}
|
||||
.status.ok { color: var(--ok); }
|
||||
.status.err { color: var(--warn); }
|
||||
|
||||
.controls {
|
||||
display: flex;
|
||||
gap: .5rem;
|
||||
margin: 1rem 0;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
button {
|
||||
background: var(--accent);
|
||||
color: #0b0d12;
|
||||
border: 0;
|
||||
padding: .6rem 1rem;
|
||||
border-radius: 6px;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
}
|
||||
button:disabled {
|
||||
background: #30363d;
|
||||
color: var(--muted);
|
||||
cursor: not-allowed;
|
||||
}
|
||||
button.danger { background: var(--warn); color: #fff; }
|
||||
|
||||
.log {
|
||||
background: #161b22;
|
||||
border: 1px solid #30363d;
|
||||
border-radius: 6px;
|
||||
padding: 1rem;
|
||||
min-height: 8rem;
|
||||
max-height: 18rem;
|
||||
overflow-y: auto;
|
||||
font-family: ui-monospace, monospace;
|
||||
font-size: .9rem;
|
||||
}
|
||||
.log .user { color: var(--accent); }
|
||||
.log .assistant { color: var(--ok); }
|
||||
.log .system { color: var(--muted); }
|
||||
|
||||
.meter {
|
||||
margin-top: 1rem;
|
||||
display: flex;
|
||||
gap: .5rem;
|
||||
align-items: center;
|
||||
}
|
||||
.meter label {
|
||||
color: var(--muted);
|
||||
font-size: .8rem;
|
||||
}
|
||||
progress {
|
||||
flex: 1;
|
||||
height: 8px;
|
||||
}
|
||||
Reference in New Issue
Block a user