omnivoice

1---2name: omnivoice3description: "Local TTS, voice cloning, voice design, and video dubbing via the VoiceStudio MCP server (open-source ElevenLabs alternative; nothing leaves the machine, runs on MPS/CUDA/CPU). Use when: (1) generating speech from text in any of 646 languages, (2) cloning a voice from a 3-second reference clip, (3) designing a voice by gender/age/accent/pitch/style, (4) dubbing a video into another language, (5) listing voice profiles or personality presets, (6) producing narration where privacy, cost, or absent API keys matter, (7) non-English narration where Edge TTS/kokoro fall short, (8) batch audio for blog posts or content pipelines. Triggers: 'omnivoice', 'voice clone', 'clone this voice', 'tts', 'narrate', 'generate speech', 'voice synthesis', 'dub video', 'voice design', 'local tts', 'multilingual voice', 'narrate this post', 'elevenlabs alternative'."4---5 6# VoiceStudio7 8The canonical cross-agent package lives at `skills/omnivoice/SKILL.md`. This9Claude-specific package retains the MCP lifecycle helpers and references.10 11## Overview12 13Generate audio locally via the VoiceStudio MCP server. Tools: `generate_speech`, `list_voices`, `list_personalities`, `list_languages`, `check_health`. Resources: `voice://{id}`, `history://recent`.14 15## Prerequisites — Backend Must Be Running16 17The MCP tools all hit `$OMNIVOICE_API_URL` (default `http://localhost:3900`). If the backend is down, every tool returns a connection error. Install + boot:18 19```bash20git clone https://github.com/debpalash/VoiceStudio.git "$OMNIVOICE_HOME"21cd "$OMNIVOICE_HOME"22uv sync23VIRTUAL_ENV="$(pwd)/.venv" uv pip install 'mcp[cli]'24```25 26Then:27 28```bash29scripts/check-health.sh        # exit 0 if up30scripts/start-backend.sh       # boot in background (MPS/CUDA auto-detected)31```32 33First synthesis call lazy-downloads the `k2-fsa/OmniVoice` model (~2.4 GB) from HuggingFace — cached on subsequent boots.34 35## Task Index — Pick the Right Tool36 37| Task | Tool | Notes |38|---|---|---|39| Verify backend is up | `check_health` | Returns `{"status":"ok","device":"mps|cuda|cpu"}` |40| Text → audio with a saved voice | `generate_speech(text, profile_id)` | Returns base64 WAV. `profile_id="demo0001"` is the bundled demo voice |41| Text → audio without a clone (voice design) | `generate_speech(text, instruct="…")` | Omit `profile_id`; pass an `instruct` like `"warm middle-aged female narrator, calm pace"` |42| Multilingual narration | `generate_speech(text, language="es")` | Any ISO 639 code or `"Auto"` |43| List existing voices | `list_voices` | Returns id, name, type, personality |44| List personality presets | `list_personalities` | Returns narrator / casual / news-anchor / etc. with their `instruct` strings |45| List supported languages | `list_languages` | 646 total; returns 20 popular + the full count |46 47For non-trivial decisions (which engine to use, when to pick VoiceStudio over kokoro / Edge TTS / ElevenLabs), see [references/engines-comparison.md](references/engines-comparison.md).48 49For MCP wiring details, backend lifecycle, troubleshooting, and a clean teardown, see [references/mcp-setup.md](references/mcp-setup.md).50 51## Common Workflows52 53### 1. One-shot narration with the demo voice54 55```python56# As called through the MCP client (your agent will do this for you):57result = generate_speech(58    text="Hello — this is VoiceStudio generating speech locally.",59    profile_id="demo0001",60    language="English",61    steps=16,                   # 8 = fast/draft · 16 = balanced · 32 = quality62)63# result is JSON with audio_id, generation_time_s, audio_duration_s, format, wav_base6464```65 66Benchmark: 4.2 s of audio in ~24 s server-side on Apple Silicon MPS at 16 diffusion steps.67 68### 2. Save the WAV to disk and play69 70Tool returns base64 PCM WAV (16-bit, mono, 24 kHz). Decode + write:71 72```python73import base64, json74payload = json.loads(result_text)            # parse JSON the tool returns75open("out.wav","wb").write(base64.b64decode(payload["wav_base64"]))76```77 78On macOS: `afplay out.wav`. Convert to MP3 with `ffmpeg -i out.wav -codec:a libmp3lame -b:a 128k out.mp3`.79 80### 3. Voice clone — end-to-end recipe81 82Cloning needs a 3-10 second reference clip the model will use as a speaker embedding. The MCP server does NOT expose profile creation — it only reads existing profiles. Two paths to create one:83 84**Path A — bundled helper (macOS, recommended for fresh clones):**85 86```bash87scripts/record-reference.sh ~/Downloads/my-ref.wav 12 188# args: output_path raw_duration_sec mic_index89# Default mic_index=1 (MacBook built-in); list devices via:90#   ffmpeg -f avfoundation -list_devices true -i ""91```92 93The script gives **audible** countdown + start/stop cues via macOS `say` + `/System/Library/Sounds/Ping.aiff` so the user knows when to speak (terminal stdout is buffered — text "speak now" prompts arrive too late). It records a longer raw window, then trims to ~10 seconds of speech via `silenceremove + atrim`, plays back for verification, and prints the next-step `curl` command.94 95**Path B — manual:**96 97```bash98# 1. Record (mono, 24 kHz native — matches model's internal rate)99ffmpeg -f avfoundation -i ":1" -t 12 -ac 1 -ar 24000 raw.wav100 101# 2. Trim leading silence + take first 10 sec of speech102ffmpeg -i raw.wav \103  -af "silenceremove=start_periods=1:start_silence=0.05:start_threshold=-40dB,atrim=end=10" \104  -ac 1 -ar 24000 ref.wav105 106# 3. Verify107ffmpeg -i ref.wav -af volumedetect -f null - 2>&1 | grep volume   # max should be > -20 dB108afplay ref.wav109```110 111**POST to /profiles** (multipart/form-data — required fields: `name`, `ref_audio`):112 113```bash114curl -X POST http://127.0.0.1:3900/profiles \115  -F "name=carlos-clone" \116  -F "ref_audio=@ref.wav" \117  -F "ref_text=The exact text spoken in the clip" \118  -F "language=English" \119  | python3 -m json.tool120# returns { "id": "abc12345", "name": "carlos-clone" }121```122 123Once created, pass `profile_id` to `generate_speech` (via MCP) or directly via `POST /generate`. Profiles persist in SQLite + reference-audio files at `~/Library/Application Support/OmniVoice/voices/<id>.<ext>` (the backend preserves the uploaded extension — `.wav` if you uploaded a WAV, `.mp3` if MP3, etc.). State persists across backend restarts.124 125**Reference clip tips that materially affect quality:**126 127| Factor | Why it matters |128|---|---|129| Single speaker | Mixed speakers blur the embedding |130| Clean speech, no music/noise | Model embeds the noise too |131| Natural prosody (avoid pangrams) | Diffusion samples replicate prosody, not just timbre |132| 3-10 sec is the sweet spot | < 3 s lacks information; > 10 s adds compute without quality gain |133| Match `ref_text` to what's spoken | Improves alignment, especially on noisy refs |134| `language` correct | Wrong language → cross-lingual transfer artifacts |135| Loudness peak ≥ -15 dB | Quiet refs work but normalize poorly |136 137### 4. Voice design (no reference clip)138 139Skip `profile_id`; provide an `instruct` string describing the desired voice:140 141```python142generate_speech(143    text="Welcome to the future of agentic systems.",144    instruct="warm middle-aged female narrator, calm authoritative pace, documentary style",145)146```147 148Get pre-made instructs via `list_personalities` and copy the one matching the brief (narrator, casual, news-anchor, etc.).149 150### 5. Video dubbing (web UI only)151 152The MCP server does not expose the dubbing endpoint. The full transcribe → translate → re-voice → mux pipeline lives behind the desktop UI (`bun run desktop` in `$OMNIVOICE_HOME`) and the `/dub/*` REST routes. When the user asks to dub a video, point them to the UI; surface this skill only for the synthesis primitives above.153 154## When NOT to use VoiceStudio155 156- **Fast English-only narration on weak hardware** → `kokoro-tts` is ~10× smaller and 2× realtime on CPU (see [references/engines-comparison.md](references/engines-comparison.md))157- **Lowest-friction one-off TTS** → Edge TTS needs no install or backend158- **Highest possible quality regardless of cost** → ElevenLabs still wins on English narration polish; VoiceStudio ties or wins on multilingual + cloning159- **Real-time streaming dictation** → use the VoiceStudio desktop widget (`⌘+⇧+Space`), not the MCP server160 161## Resources162 163- [references/engines-comparison.md](references/engines-comparison.md) — Decision tree across VoiceStudio / kokoro / Voicebox / Edge TTS / ElevenLabs / cloud APIs164- [references/mcp-setup.md](references/mcp-setup.md) — MCP wiring, backend lifecycle, env vars, troubleshooting165- [scripts/check-health.sh](scripts/check-health.sh) — `curl /health`, exit 0/1166- [scripts/start-backend.sh](scripts/start-backend.sh) — Start uvicorn on 127.0.0.1:3900 with health probe167- [scripts/stop-backend.sh](scripts/stop-backend.sh) — Clean shutdown via `kill -TERM` on the bound PID168- [scripts/record-reference.sh](scripts/record-reference.sh) — macOS-only: record + trim + verify a reference clip for cloning, with audible cues (`say` + system beeps) that bypass terminal output buffering169 170Backend Swagger / OpenAPI: `http://127.0.0.1:3900/docs` (when backend is up).171 172Upstream: github.com/debpalash/VoiceStudio. The app uses AGPL-3.0-only; optional engines and downloaded models retain their own licenses. See `LICENSE-NOTICE.md` in the repository.173