🦜 VieNeu-TTS

VieNeu-TTS is the next generation of on-device Vietnamese TTS: 10,000+ hours of bilingual training, instant voice cloning, a dedicated Podcast/Conversation mode, and extremely fast — real-time streaming with first audio in ~115 ms, RTF ≈ 0.01–0.02 batched on a consumer GPU (RTX 3060), ~2× real-time on a plain CPU (benchmarks).

[!IMPORTANT] 🦜 VieNeu-TTS v4 — available on vieneu.io

VieNeu-TTS v4 delivers near-original voice cloning fidelity, allowing a short reference clip to be reproduced with very high speaker similarity.

Due to the strength of its voice-cloning capabilities and the potential for misuse, v4 is proprietary and will not be open-sourced. It is available exclusively through the VieNeu API / vieneu.io.

VieNeu-TTS v3 Turbo remains the latest open-source version available in this repository. Future open-source releases, including potential v3.x updates, may also be published here.

[!NOTE] 🦜 VieNeu-TTS v3 Turbo is officially released! A brand-new architecture designed and trained from scratch by Phạm Nguyễn Ngọc Bảo (codec: MOSS-Audio-Tokenizer-Nano; phonemizer: sea-g2p): - 48 kHz high-fidelity audio (up from 24 kHz). - 25 built-in preset voices (10 editors' picks) — stable and consistent, no reference clip needed. - Natural reading style everywhere — the style follows the reference voice (the style argument is deprecated and ignored). - Emotion / non-verbal cues (experimental): drop [cười], [thở dài], [hắng giọng] straight into the text. - Batched generation (batch size up to 32), including a multi-speaker Conversation mode that batches the whole script regardless of speaker. - Instant voice cloning from a 3–8s clip, with automatic reference denoising. - Real-time streaming + OpenAI-compatible API + Docker — POST /v1/audio/speech drop-in for the OpenAI SDK / Pipecat / LiveKit; first audio in ~115 ms and 16 concurrent streams under 200 ms on a single RTX 3060 (32 max), CPU-only streaming too. See §3, §4 Benchmarks and docs/streaming.md. - LoRA fine-tuning — train your own voice or reading style on one consumer GPU (§5).

Try it in the Web UI (backbone "VieNeu-TTS-v3-Turbo") or the SDK (Vieneu(mode="v3turbo"), the default).

🎬 Demos

[!TIP] Dubbing, Lecture, Tiktok Reel and Audiobook features are exclusively available in the official VieNeu App. This GitHub repository only provides a simplified Gradio demo interface and the core SDK for developers.

📌 Table of Contents

1. 🦜 Installation & Web UI
2. 📦 Using the Python SDK
3. 🐳 API Server & Docker — OpenAI-compatible streaming API (v3 Turbo) · legacy v2 server
4. 📊 Benchmarks — every speed / latency number in one place (CPU vs GPU, batch, streaming, Nano)
5. 🎓 Fine-tuning (LoRA)
6. 🔬 Model Overview
7. 🚀 Roadmap
8. 🤝 Support & Contact
9. 📑 Citation

🦜 1. Installation & Web UI

[!TIP] On Windows? The fastest way to get started is the standalone installer at vieneu.io/#/download — no need to install uv or clone the repo manually. macOS: a similar installer is coming in an upcoming release; for now, please use the uv sync steps below. Docker? Skip the steps below: --profile api-gpu / api-cpu (OpenAI-compatible streaming API) — see §3 API Server & Docker.

Setup with uv (Recommended)

uv is the fastest way to manage dependencies.

# Windows:
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# Linux/macOS:
curl -LsSf https://astral.sh/uv/install.sh | sh

1. Clone the Repo: bash git clone https://github.com/pnnbao97/VieNeu-TTS.git cd VieNeu-TTS

2. Install Dependencies:

   📊 Which one? CPU ≈ RTF 0.5 (2× real-time, one stream) · GPU ≈ RTF 0.02 batched (~50× real-time) and 16 real-time streams — all measurements in §4 Benchmarks.

• Option 1: CPU & macOS (minimal, torch-free) — RTF ≈ 0.5, no GPU needed

  bash uv sync - Option 2: GPU — v3 Turbo on GPU (PyTorch) — RTF ≈ 0.02 batched, 16 real-time streams

  bash uv sync --extra cuda

3. Start the Web UI: bash uv run vieneu-web Access the UI at http://127.0.0.1:7860.

📦 2. Using the Python SDK (vieneu)

The vieneu SDK defaults to VieNeu-TTS v3 Turbo (48 kHz). The minimal install is torch-free: on CPU everything runs on ONNX Runtime (PyTorch is never imported), and on a CUDA machine it auto-switches to the PyTorch engine — where inference is batched automatically (same API, no code change).

Quick Start

CPU (default) — torch-free, runs v3 Turbo via ONNX Runtime. Most users want this — RTF ≈ 0.5 on a 12th-gen Core i5 (numbers in §4 Benchmarks):

⚡On CPU the backbone runs fp32 by default (maximum fidelity). Need more speed? Pass Vieneu(precision="int8") — ~1.6× faster (RTF ≈ 0.35) and ~4× smaller, but it requires a CPU with VNNI (AVX-512 VNNI / AVX-VNNI); on older CPUs int8 can produce garbled audio. precision only affects the CPU/ONNX path; on GPU it's ignored (PyTorch).

🪶 Still too slow, or deploying on a phone / ARM board? Try VieNeu-TTS v3 Nano (preview) — Vieneu(mode="v3nano"), ~3× faster than Turbo fp32 on CPU (RTF 0.11–0.22 on a desktop CPU), but noticeably lower quality (especially English / bilingual), 24 kHz, 11 preset voices + voice cloning. Details and caveats in the v3 Nano section below.

pip install vieneu

GPU (CUDA) — only if you have an NVIDIA GPU; ~25× faster than the CPU path (RTF ≈ 0.02 batched, 16 real-time streams — §4 Benchmarks). On Linux pip install "vieneu[cuda]" is enough (PyPI torch ships CUDA there); on Windows install the CUDA torch first as below.

⚡ Every audio frame on GPU is one CUDA graph (acoustic decoder + sampling + repetition penalty + backbone step in a single replay — no torch.compile, no C++ toolchain needed). The first call for each batch size pays ~0.5 s to capture the graph (kept afterwards; servers can call warm_fused() at start-up). VIENEU_FUSED_FRAME=0 restores the plain loop.

pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install "transformers==4.57.6"   # pinned — most stable transformers for the GPU SDK
pip install vieneu

```python import time from vieneu import Vieneu

Default = v3 Turbo (48 kHz). GPU → PyTorch (auto-detected).

vieneu = Vieneu() # On a GPU machine you can still switch to ONNX/CPU if you prefer: Vieneu(backend="onnx")

1. Built-in voice by name — no reference clip needed

print("🔊 Generating speech...")

start_time = time.time() audio = vieneu.infer("[cười] Trời ơi, cái giọng nó tự nhiên mà nó mượt mà dã man, nghe không khác gì người thật luôn. Giờ thì tha hồ mà quẩy content với cả kho giọng nói đa dạng, đủ mọi sắc thái biểu cảm. Mọi người bật loa lên rồi cùng trải nghiệm thử với mình nhé!", voice="Phạm Tuyên") elapsed_time = time.time() - start_time

vieneu.save(audio, "output.wav") print("✅ Saved to output.wav")

Tính RTF (Real-Time Factor)

sample_rate = 48000 audio_duration = len(audio) / sample_rate rtf = elapsed_time / audio_duration

print(f"\n⏱️ Thời gian xử lý: {elapsed_time:.3f}s") print(f"🎵 Thời lượng audio: {audio_duration:.3f}s") print(f"📊 RTF: {rtf:.4f} ({'nhanh hơn' if rtf < 1 else 'chậm hơn'} real-time {1/rtf:.2f}x)" if rtf > 0 else "")

List the built-in voices

voices = vieneu.list_preset_voices() print(f"\n🎙️ {len(voices)} built-in voices available:") for label, voice_id in voices: print(f" - {label} ({voice_id})")

2. ⚡ Batch on GPU: infer_batch() runs many texts in ONE batched forward — same API.

On a CUDA GPU the chunks from every text share each forward step (big throughput

win). On CPU it still WORKS (no error) — just sequentially, so there's no batch

gain. Batch caps at max_batch_size (default 32; tune via Vieneu(max_batch_size=64)

or infer_batch(..., batch_size=64),