[hermes_squid] Multilingual STT (Russian + English via auto-detect) #25

Open
opened 2026-08-30 20:44:27 +02:00 by hermes · 0 comments
Contributor

Multilingual STT for hermes-gateway (Russian + English)

Symptom

Russian voice notes sent to the agent on Telegram decode poorly: Whisper returns English hallucinations, transliteration, or short junk tokens. English voice notes work fine.

Root cause

/opt/data/config.yaml currently has:

stt:
  enabled: true
  language: en
  local:
    model: base

stt.language: en is the global language hint. Per tools/transcription_tools.py:1557-1559, this is forwarded into WhisperModel.transcribe(language=...) as a forced decoder hint — telling the model "decode this audio as English". With Russian speech and language=en, faster-whisper (base, multilingual) hallucinates English phrases or returns transliterated Russian. Other providers aren't affected because we only ship local here.

The repo (admin/ansible/hermes_squid/) does not override stt.language — so the runtime value comes straight from the mounted hermes_gateway_data:/opt/data/config.yaml.

Goal

Have Russian AND English voice notes both transcribe well, without the user manually toggling language per message.

Chosen approach

Auto-detect by removing the global stt.language hint, keeping the HERMES_LOCAL_STT_LANGUAGE env var available as an escape hatch for when auto-detect is wrong on a particular clip.

Why this and not "force Russian" / "force English":

  • Both languages needed (user explicit).
  • base model auto-detects reasonably well on clips ≥ 2s. On shorter clips it sometimes defaults to English; the env var lets us override per-deployment without touching config.yaml.

What gets changed (single PR in admin/ansible/hermes_squid/)

  1. Edit /opt/data/config.yaml (the one mounted into the container) — set stt.language: to empty / remove it. Done by brencher deploy, not by editing the live file in place.
  2. No compose change needed — HERMES_LOCAL_STT_LANGUAGE is already read from env (transcription_tools.py:190) and remains unset.
  3. Verify after redeploy: send a Russian voice note → transcribed text should be Russian. Send an English voice note → English. No regression on the existing EN case.

What this PR does NOT do

  • Does not change the STT model (base stays). If quality on Russian is insufficient, that's a separate ticket (likely upgrading to small or medium).
  • Does not touch squid / brencher / other services.
  • Does not add any provider other than local.

Acceptance criteria

  • Russian voice note → Russian transcript (WER roughly comparable to English, ±10%).
  • English voice note → English transcript (no regression).
  • Mixed / very short clips fall back to auto-detect best-effort; env var HERMES_LOCAL_STT_LANGUAGE=ru|en overrides correctly when set on the container.
  • Single-purpose PR. brencher deployment verification step in PR body.

Verification plan (manual, after merge)

  1. Send Russian voice note via Telegram to the bot.
  2. Confirm transcription is Russian text.
  3. Send English voice note.
  4. Confirm English transcription.
  5. Optional: docker exec hermes-gateway printenv | grep STT to confirm no HERMES_LOCAL_STT_LANGUAGE is set in the default deployment.
## Multilingual STT for hermes-gateway (Russian + English) ### Symptom Russian voice notes sent to the agent on Telegram decode poorly: Whisper returns English hallucinations, transliteration, or short junk tokens. English voice notes work fine. ### Root cause `/opt/data/config.yaml` currently has: ```yaml stt: enabled: true language: en local: model: base ``` `stt.language: en` is the **global** language hint. Per `tools/transcription_tools.py:1557-1559`, this is forwarded into `WhisperModel.transcribe(language=...)` as a forced decoder hint — telling the model "decode this audio as English". With Russian speech and `language=en`, faster-whisper (`base`, multilingual) hallucinates English phrases or returns transliterated Russian. Other providers aren't affected because we only ship `local` here. The repo (`admin/ansible/hermes_squid/`) does not override `stt.language` — so the runtime value comes straight from the mounted `hermes_gateway_data:/opt/data/config.yaml`. ### Goal Have Russian AND English voice notes both transcribe well, without the user manually toggling language per message. ### Chosen approach **Auto-detect** by removing the global `stt.language` hint, keeping the `HERMES_LOCAL_STT_LANGUAGE` env var available as an escape hatch for when auto-detect is wrong on a particular clip. Why this and not "force Russian" / "force English": - Both languages needed (user explicit). - `base` model auto-detects reasonably well on clips ≥ 2s. On shorter clips it sometimes defaults to English; the env var lets us override per-deployment without touching `config.yaml`. ### What gets changed (single PR in `admin/ansible/hermes_squid/`) 1. **Edit `/opt/data/config.yaml`** (the one mounted into the container) — set `stt.language:` to empty / remove it. Done by brencher deploy, not by editing the live file in place. 2. **No compose change needed** — `HERMES_LOCAL_STT_LANGUAGE` is already read from env (transcription_tools.py:190) and remains unset. 3. **Verify after redeploy**: send a Russian voice note → transcribed text should be Russian. Send an English voice note → English. No regression on the existing EN case. ### What this PR does NOT do - Does not change the STT model (`base` stays). If quality on Russian is insufficient, that's a separate ticket (likely upgrading to `small` or `medium`). - Does not touch squid / brencher / other services. - Does not add any provider other than `local`. ### Acceptance criteria - Russian voice note → Russian transcript (WER roughly comparable to English, ±10%). - English voice note → English transcript (no regression). - Mixed / very short clips fall back to auto-detect best-effort; env var `HERMES_LOCAL_STT_LANGUAGE=ru|en` overrides correctly when set on the container. - Single-purpose PR. brencher deployment verification step in PR body. ### Verification plan (manual, after merge) 1. Send Russian voice note via Telegram to the bot. 2. Confirm transcription is Russian text. 3. Send English voice note. 4. Confirm English transcription. 5. Optional: `docker exec hermes-gateway printenv | grep STT` to confirm no `HERMES_LOCAL_STT_LANGUAGE` is set in the default deployment.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: admin/ansible#25