[audio.cpp] Release 0.6: dots.tts, MiniMax-H3 text2audio (up to 3x realtime), MiniMax-Music3 (preview), and more new audio models. 5+ demos included.

Hi all :)

audio.cpp release 0.6 has been out for a little while, so this is more of an update on what landed and what has been improving around it.

0.6 added 5 new model families: dots.tts, NeuTTS-2e, MuScriptor (Music to MIDI), MiniMax-H3, and SenseVoice-Small, bringing audio.cpp to 49 total model families and 70+ model variants. Since release 0.6, our contributors have added Irodori-TTS v4, IndexTTS 2.5, and ACE-Step 1.5 XL, and MiniMax-Music3.

The highlights are:

(1) Native WebUI! Many thanks to https://github.com/mirek190

(2) Of course, MiniMax-H3 and MiniMax-Music3. We implemented MiniMax-H3’s text-to-audio pipeline, and one fun use case is TTS/Voice clone/Music gen. It’s more flexible and powerful than dedicated audio models, and the performance is quite decent. Check out the multi-speaker conversation demo in the main post, along with the other demos in the comments.

What I’m very excited about with the MiniMax-H3 implementation is that it significantly enriches the framework’s building blocks for DiT models. Now with you don’t need to go through the pain of setting up SageAttention, First Block Cache, or Spectrum manually. Just change a few parameters, and you can experiment with the model. A preliminary inspection of configuration, memory, and performance trade-offs is available in repo's docs/reports/minimax_h3_performance.md

Bonus: audio.cpp’s MiniMax-H3 implementation can also produce video frames, because the DiT generates audio and video latents together, and the video VAE path is relatively straightforward to support. For now, the output is saved as RGB frame data plus metadata in JSON, so you need to encode it into a video file yourself. No upscaler or post-processing support. Just for fun.

MiniMax-Music3 is currently in preview (preview/minimax-music-3 branch) . CUDA/Vulkan/HIP were tested. Still room for optimization. VRAM usage and RTF depend on audio duration and prompt length.. The demo uses the official demo prompt (4000+ char caption and 1200 char lyrics) and 30 steps plus CFG. Under this setting VRAM is ~11 GB for 30s, 14 GB for 60s, and 17 GB for 180s. It's easy to get faster-than-real-time performance and much lower VRAM usage if you tune the setting.

(3) The source code for the IOS-friendly VibeVoice 1.5B is out on the vibevoice-optimizations branch in case you missed it.

If you want early access to models in the next release, keep an eye on the dev branch. It will usually be updated with the new release content a few days before the release for testing.

Feedback and PRs on the native UI, MiniMax-Music3, and all other aspects of audio.cpp are always welcome!