OpenVoice for Unity

A companion with several avatar characters needs several voices, and each voice has to carry emotion. Doing that in the cloud means a network round-trip for every sentence. In this prototype we built the voices on OpenVoice V2 and split the work so the heavy part happens once, at development time, and only light inference runs on the phone.

Authoring on the workstation

Our Python tools take a clean reference recording for each voice and extract two kinds of embedding:

• a tone color embedding that captures the voice's timbre, for cloning;
• style embeddings for each affect variant (neutral, warm, melancholic, tense).

The tools validate the embeddings, report how similar the voices and styles are to one another, and pack everything into a small binary bundle that ships inside the app. The same toolchain exports the model components from PyTorch to ONNX for every platform and converts them to Core ML for iOS, with quantization.

Inference in Unity

A C# backend interface hides the platform. On iOS, a Swift service runs the Core ML models and reaches Unity through a native bridge. On Windows and macOS, a C# backend runs the ONNX models. On top sits a TTS manager and an affect coordinator that picks the style variant for each sentence from upstream sentiment, so the voice can match an avatar's expression. Editor tools validate the setup and patch the iOS build.

Swapping the backbone

OpenVoice's default backbone, MeloTTS, is multilingual. Its shared phoneme and tone tables pull English vowels slightly off native American English, and it also needs a large BERT model at inference. The repo carries the Piper engine from PiperTTS for Unity so a monolingual Piper voice can stand in as the backbone while OpenVoice's tone color converter keeps the cloning. That drops the BERT model and shrinks the footprint considerably.

This was part of our on-device voice research for Frndo. The app's voice pipeline uses a different, server-side approach, and its avatars are animated by its own lip-sync system.