CosyVoice 3: Free Open-Source AI Voice Cloning That Runs on CPU
Clone a voice from just 8 seconds of audio, even across languages. Free, open-source voice cloning on CPU: setup, tests, and instructions.

In my last post, Kokoro TTS vs Chatterbox TTS, I compared a handful of open-source TTS (text-to-speech) models and promised to cover each one in more detail. This one is about CosyVoice 3. Give it an 8-second recording of someone, and it will say brand-new sentences in that person's voice. It even speaks Chinese dialects like Cantonese and Sichuanese.
My laptop doesn't have an NVIDIA GPU, and it still runs on the CPU alone. One line takes about two minutes, so it works best when you queue up a whole batch of lines and let it grind away in the background. If your machine has Intel integrated graphics, you can use that instead and get roughly a 5x speedup. Below, I cover the full installation process, every snag I hit, and a voice cloning demo using my own voice.
What is CosyVoice 3? Can you use it commercially?
CosyVoice is an open-source speech generation model from Alibaba (the source code's copyright notice reads Alibaba Inc.). The latest open release is Fun-CosyVoice3-0.5B-2512, published in December 2025. The details below come from the Hugging Face model page, checked on 2026-11-21:
Item | Details |
|---|---|
Parameters | 0.5B (500 million) |
License | Apache-2.0, commercial use allowed |
Languages | Chinese, English, Japanese, Korean, German, Spanish, French, Italian, Russian (9 in total) |
Chinese dialects | 18+, including Cantonese, Minnan, Sichuanese, Northeastern Mandarin, and Shanghainese |
Voice cloning | Clone from a single reference clip, including across languages (a Chinese voice speaking English) |
Instruction control | Set the dialect, speed, volume, or emotion with one sentence |
In the official benchmark table, CosyVoice 3 has a lower character error rate (CER) on the Chinese test set than its predecessor, CosyVoice 2, as well as higher speaker similarity.
Is CosyVoice 3.5 open source?
No. Alibaba announced CosyVoice 3.5 in March 2026, but so far it's only available through the API on Alibaba Cloud Model Studio, and only in the Beijing region. As of 2026-11-21, there are no 3.5 weights on Hugging Face. A user asked on GitHub whether it will be released as open source, but the team hasn't replied. If you want local voice cloning with a commercial license, 3.0 is what you can use today.
Also note that the CosyVoice GitHub repo has moved from FunAudioLLM/CosyVoice to QwenAudio/CosyVoice. The old URL redirects automatically.
Hardware: can it run without an NVIDIA GPU?
Yes, just more slowly. My laptop has an Intel Core Ultra 5 322 (6 cores) with Panther Lake integrated graphics and no discrete GPU. Here are the results for the same Chinese line, about 8 seconds long:
Hardware | Generation time | Real-time factor (lower is faster) |
|---|---|---|
CPU (6 cores) | 115 s | 15.6 |
Intel iGPU (PyTorch XPU) | 21.5 s | 2.5 |
Loading the model takes another 12 seconds or so. When I generate a batch on the iGPU, the model loads once, and each 4- to 6-second line finishes in about 15 to 30 seconds.
The official installation process is built around NVIDIA GPUs (CUDA). Using an Intel iGPU takes a few extra steps, which I cover further down.
How to install CosyVoice 3 (CPU version)
I'm on Linux (Arch) and manage Python with uv.
1. Get the code
mkdir -p ~/tools/cosyvoice && cd ~/tools/cosyvoice
git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git repoDon't skip --recursive: Matcha-TTS, which CosyVoice depends on, is a submodule.
2. Create the environment and install packages
The official requirements.txt pulls in the CUDA build of PyTorch, TensorRT, and DeepSpeed. My laptop has no NVIDIA GPU, so none of that is useful. I installed only what's actually needed, with the CPU build of PyTorch:
uv venv --python 3.10 .venv
VIRTUAL_ENV=.venv uv pip install "setuptools<81" wheel
VIRTUAL_ENV=.venv uv pip install --no-build-isolation-package openai-whisper \
--index-strategy unsafe-best-match --extra-index-url https://download.pytorch.org/whl/cpu \
"torch==2.3.1+cpu" "torchaudio==2.3.1+cpu" numpy==1.26.4 conformer==0.3.2 diffusers==0.29.0 \
hydra-core==1.3.2 HyperPyYAML==1.2.3 inflect==7.3.1 librosa==0.10.2 omegaconf==2.3.0 \
onnx==1.16.0 onnxruntime==1.18.0 openai-whisper==20231117 protobuf==4.25 pyworld==0.3.4 \
soundfile==0.12.1 transformers==4.51.3 x-transformers==2.11.24 wetext==0.0.4 modelscope==1.20.0 \
lightning==2.2.4 matplotlib==3.7.5 rich==13.7.1 pyarrow==18.1.0 networkx==3.1 \
wget==3.2 gdown==5.1.0 pydantic==2.7.0 huggingface_hubI ran into three problems here:
openai-whisperfails to build: its installer needspkg_resources, which newer setuptools releases have dropped. Installsetuptools<81first, then tell uv not to build it in an isolated environment (--no-build-isolation-package).- Without
lightningandrich, the model fails to load: the model config uses both, so install them too. - The
no frontend is avaliablemessage: the text normalization package wetext tries to download data from ModelScope, and my request was refused (403). Generation still works without it, and CosyVoice 3 handles ordinary numbers and symbols on its own. I test that later.
3. Download the model (and save 4 GB)
The full model download is about 9.8 GB, but this setup doesn't use a few of the files, so you can skip them:
from huggingface_hub import snapshot_download
snapshot_download("FunAudioLLM/Fun-CosyVoice3-0.5B-2512",
local_dir="models/Fun-CosyVoice3-0.5B",
ignore_patterns=["flow.decoder.estimator.fp32.onnx", # for TensorRT
"speech_tokenizer_v3.batch.onnx", # for batch inference
"llm.rl.pt"]) # RL version, not loaded by defaultThat leaves 5.1 GB on disk.
4. Generate your first line
CosyVoice 3 prompts follow a fixed format: they start with You are a helpful assistant.<|endofprompt|>. Adapted from the official example.py:
import sys
sys.path[:0] = ["repo", "repo/third_party/Matcha-TTS"]
import torch, torchaudio
from cosyvoice.cli.cosyvoice import AutoModel
m = AutoModel(model_dir="models/Fun-CosyVoice3-0.5B")
SYS = "You are a helpful assistant.<|endofprompt|>"
# Voice cloning: reference audio + transcript of the reference audio
out = m.inference_zero_shot("Line to speak", SYS + "What the reference audio says", "ref.wav", stream=False)
wav = torch.cat([x["tts_speech"] for x in out], dim=1)
torchaudio.save("out.wav", wav, m.sample_rate)The three main functions:
Function | What it's for |
|---|---|
| Voice cloning with the closest match. Requires a transcript of the reference audio |
| No transcript needed. Good for cross-lingual work, like a Chinese voice speaking English |
| Set the dialect, speed, or emotion with one sentence. Instruction format below |
Instructions are written like "You are a helpful assistant. 请用广东话表达。<|endofprompt|>": the same fixed opener, then the instruction, then <|endofprompt|>.
I wrapped all of this into a single command that lives in ~/tools/cosyvoice/, so every project can call it directly.
Advanced: 5x faster on Intel integrated graphics (PyTorch XPU)
PyTorch's backend for Intel graphics is called XPU. The PyTorch 2.3.1 build that CosyVoice uses doesn't support XPU yet, so I built a separate environment and left the CPU one alone.
1. Install Intel's compute runtime
A normal desktop install only ships the display driver. For PyTorch to run computations on the GPU, you also need Intel's compute runtime (Level Zero). On Arch Linux:
sudo pacman -S intel-compute-runtime2. Install the XPU build of PyTorch
uv venv --python 3.10 .venv-xpu
VIRTUAL_ENV=.venv-xpu uv pip install --index-url https://download.pytorch.org/whl/xpu torch torchaudio
.venv-xpu/bin/python -c "import torch; print(torch.xpu.is_available(), torch.xpu.get_device_name(0))"If it prints True Intel(R) Graphics, the iGPU has been detected. I ended up with PyTorch 2.14.1.
Next, install the packages listed earlier into this environment, adding -c with a constraints file. Pin the torch version you just installed in that file, so installing other packages can't swap torch out for a non-XPU build.
3. Remove the conflicting triton package
After that, import transformers failed with cannot import name 'make_tensordesc_args'. The cause: openai-whisper had pulled in an old triton, which overwrote the triton-xpu the XPU build of PyTorch needs. Uninstall the old one and reinstall triton-xpu:
VIRTUAL_ENV=.venv-xpu uv pip uninstall triton
VIRTUAL_ENV=.venv-xpu uv pip install --reinstall-package triton-xpu \
--index-url https://download.pytorch.org/whl/xpu triton-xpu4. Move the model to the iGPU
CosyVoice's code only knows about CUDA: without an NVIDIA GPU, it falls back to the CPU. Instead of patching the official code, I load the model on the CPU first and then move the three main networks to the iGPU:
m = AutoModel(model_dir="models/Fun-CosyVoice3-0.5B")
dev = torch.device("xpu")
for name in ("llm", "flow", "hift"):
getattr(m.model, name).to(dev)
m.model.device = dev
m.frontend.device = devThat way, updating the official code later won't cause conflicts. The first run on the iGPU requires compilation, which adds a dozen or so seconds. After that, it runs normally.
Test 1: cloning my own voice
I recorded 8 seconds of my voice on my phone and used it as the cloning reference.
Notice: every audio sample in this post is either my own voice (Terry L) or a clone of it. All rights reserved. Do not download, repost, or use these clips anywhere else, including to clone my voice.
I have never said the sentence below. CosyVoice 3 generated it in English from the recording above:
To me, the generated voice sounds remarkably like my own.
Cross-lingual cloning: the English line above already came from a Chinese recording. The same reference clip can also be used to generate speech in Japanese or Cantonese:
I ran the English, Japanese, and Cantonese clips through speech recognition (Whisper) and compared the transcripts. All three were correct, with no extra or missing words.
How to record the reference clip
- 5 to 10 seconds is enough. I also tried a 20-second reference, and generation took 2 to 3 times longer.
- Quiet room, one speaker: background music and noise get cloned too.
- Include a transcript: with
inference_zero_shot, the transcript has to match the recording word for word. If you don't know what's said, transcribe it with Whisper first, or switch toinference_cross_lingual. - Boost quiet recordings first: my phone recording averaged only -31 dB, so I trimmed the silence at both ends and raised the volume before cloning.
Tone and pace get cloned too
CosyVoice 3 clones more than the timbre. Speaking rate, tone, and pitch contour all follow the reference recording. With inference_zero_shot, the reference audio and transcript are treated as "what was just said," and the model continues from there, so it maintains the same delivery.
In my test, the original recording ran at about 4.3 Chinese characters per second, and the cloned Chinese line came out at about 4.4—a very close match. It isn't always that consistent, though: the same line with a different seed slowed to 3.6 characters per second, and Arthur's cloned line came out a bit faster than the reference. For important lines, generate a few versions and pick the best.
What this means in practice:
- Record the tone you want: for an angry line, record an angry reference; for something gentle and slow, read the reference slowly. One character can have several references, such as normal speech, excited speech, and whispering.
- Flaws in the reference get copied too: if it's rushed, flat, echoey, or noisy, the clone will be the same.
- Three ways to fine-tune: adjust speed directly with the
speedparameter; give an instruction throughinference_instruct2, such as "请用尽可能慢地语速说一句话。" (speak as slowly as possible); or switch toinference_cross_lingual.
The source code shows that inference_cross_lingual doesn't pass the reference audio to the language model that decides rhythm. The reference only sets the timbre, so the output is less likely to copy its pacing.
Test 2: voicing Arthur, the robot bartender
The 3D lunar base on my home page is a small game where you walk an astronaut around the base, and it has a bar run by a robot bartender named Arthur. He's a tribute to Arthur, the android bartender in the movie Passengers (2016), and a few of his lines are adapted from the film. Michael Sheen played Arthur in the movie; I deliberately avoid imitating his voice.
I continue to use my own voice (the same 8-second recording) for Arthur's lines in this section:
If a word comes out wrong, you can fix it right in the script. For Chinese characters with more than one reading, write the pinyin with a tag like [h][áng], and for English words you can specify the pronunciation with CMU phonemes.
It happened to me once: in the line used in the "breath and laughter tags" test further down, the character 尷 was mispronounced the first time. I replaced it with [g][ān], regenerated, and it came out right.
Convert Japanese to katakana first
The official example specifically says: convert Japanese to katakana before you input it. I tried both ways. With kanji and hiragana, the speech recognition transcript came back quite different, with several words mispronounced. In katakana, it was correct.
For Japanese voice work, have an AI coding assistant convert your lines to katakana first, then hand them to CosyVoice.
Test 3: controlling dialect, emotion, and speed with instructions
inference_instruct2 takes a one-sentence instruction. The official code lists the supported instructions, including 17 dialects plus volume, speed, emotion, and "speak like a robot."
Write instructions in Simplified Chinese or English. Most of the official instructions are in Simplified Chinese, and the two volume instructions are in English ("Please say a sentence as loudly as possible." and "Please say a sentence in a very soft voice.").
I tried three ways of requesting the slowest possible speed for the same line:
Instruction | Audio duration |
|---|---|
No instruction | About 4 to 5 s |
Simplified Chinese "请用尽可能慢地语速说一句话。" | 8.2 s |
English "Please say a sentence as slowly as possible." | 8.5 s |
Traditional Chinese "請用盡可能慢地語速說一句話。" | 5.3 s |
Simplified Chinese and English both noticeably slowed it down; Traditional Chinese barely made a difference. The English instruction isn't on the official list, but it works anyway, so if you don't read Chinese, just write your instructions in English.
All of the clips below use my voice. The line is "歡迎來到月球基地的酒吧。今天想喝點什麼?" ("Welcome to the lunar base bar. What can I get you today?"):
For Cantonese, the line needs to be written in Cantonese (嚟, 嘅, 啲, 咩) for the speech recognition transcript to come back fully correct.
Can CosyVoice speak Hokkien?
The official dialect list includes Minnan (Hokkien). I tried "请用闽南话表达。" on the same line, and to my ear, it sounded nothing like Taiwanese Hokkien: the intonation was off throughout.
Breath and laughter tags
You can add tags like [breath] (a breath), <laughter>…</laughter> (speaking while laughing), and <strong>…</strong> (emphasis) to a line:
Honestly, I couldn't hear the breath or the laughter at all. At least in this test, the tags had no noticeable effect, so don't expect too much.
Numbers and times
As mentioned earlier, my setup is missing the text normalization package. Here's a test line with numbers and a time:
"120," "8 折" (20% off), and "23:30" were all read correctly. Even without the text normalization package, CosyVoice 3 handles ordinary numbers and times.
For trickier formats like phone numbers, version numbers, dates, or times, it's still safer to spell them out in words. In Chinese, for example, write a time as "十一點半" (half past eleven).
CosyVoice 3 vs. Qwen TTS and other open-source TTS models
Compared with the models in my previous post, here's what stands out about CosyVoice 3:
- Much better Chinese cloning than Chatterbox's Chinese version: I ran Chatterbox's Chinese version with the same reference clip and the same lines. Many words came out mumbled or wrong, and the quality was poor. If you need to clone a Chinese voice, skip Chatterbox and save yourself the testing.
- Voice cloning plus instruction control: CosyVoice 3 lets you set the dialect, emotion, and speed with one sentence. Kokoro only has built-in voices and can't clone.
- No need to convert the whole script: when a word is mispronounced, you fix just that word with a pinyin tag, instead of converting the entire line to Simplified Chinese the way Qwen3-TTS requires.
- Fewer languages: only 9. Chatterbox's multilingual version supports 23; languages like Vietnamese, Indonesian, and Turkish aren't supported by CosyVoice 3.
- Much slower than Kokoro: Kokoro generates a line in a dozen or so seconds on the CPU. CosyVoice 3 only becomes practical on an iGPU.
Can you clone someone else's voice?
Technically, anyone's voice can be cloned from a clean recording, and the results are good. That's exactly why you should only clone your own voice or a voice whose owner has given you permission. Celebrities' voices are protected by personality rights, and making someone say things they never said can also amount to fraud or defamation.
CosyVoice 3's Apache-2.0 license allows commercial use of the model itself. It doesn't give you the right to use anyone's voice. My approach: use my own voice, or generate a character's base voice using one of the built-in voices in an open-source model like Kokoro, then clone that.
FAQ
Is CosyVoice 3 free for commercial use?
Yes. Fun-CosyVoice3-0.5B-2512 is licensed under Apache-2.0, which allows commercial use. But the reference voice has to be your own or used with the owner's permission.
Can I clone my own voice for free with CosyVoice 3?
Yes. The model is free and open source, and it runs locally on your own computer. In my test, an 8-second phone recording plus its transcript was enough, and the clone sounded very close to my real voice.
Does CosyVoice 3 speak Cantonese or Hokkien?
Cantonese, yes, and it works best when the line is written in Cantonese (嚟, 嘅, 啲). The official dialect list includes Minnan (Hokkien), but in my test it didn't sound like Taiwanese Hokkien, and the intonation was off, so it isn't suitable for Hokkien voice work right now.
Can I run CosyVoice 3 without an NVIDIA GPU?
Yes. On the CPU, an 8-second line takes almost two minutes; on Intel integrated graphics (PyTorch XPU), it's about 20 seconds. You'll need to install Intel's compute runtime and the XPU build of PyTorch first.
Is CosyVoice 3.5 open source?
Not as of 2026-11-21. CosyVoice 3.5 is only available through the Alibaba Cloud Model Studio API, and its weights aren't on Hugging Face.
How long should a voice cloning reference clip be?
5 to 10 seconds is enough. Record in a quiet place with only one person speaking, and include a word-for-word transcript. The longer the reference, the slower the generation.
Wrapping up: your own voice-over from an 8-second clip
With CosyVoice 3, using my own voice for my games and videos finally feels doable: record 8 seconds, then let the model read every line after that, even in English, Japanese, or Cantonese. No NVIDIA GPU? No problem: it runs on the CPU, and it's 5x faster with Intel integrated graphics.
If you've tried CosyVoice 3 or want to see tests of other models, come say hi on Threads or X.
