Kokoro-82M: CPU and GPU benchmarks
tl;dr ¶
- I benchmarked the Kokoro-82M text-to-speech model on CPU and GPU, using the same Nix package for both.
- The CPU is an Intel Core i7-11700 (8 cores / 16 threads, 2.5–4.9 GHz).
- The GPU is the one this machine exposes: NVIDIA GeForce RTX 3060 Laptop GPU, 6 GB, 80 W, according to
nvidia-smi. - Warm synthesis is 5.08× faster at 5 words, 7.3–8.7× at 20–100 words, and 10.1× at 300–1000 words.
- The GPU holds a steady ~32× realtime; the CPU does 3.2–4.2× realtime.
- A fresh one-sentence command is slightly slower on the GPU (0.86×) because CUDA session setup costs 1.5–2.4 seconds. A 57-word paragraph is already 2.79× faster.
- Phonemization is not the bottleneck: espeak takes 0.02 seconds for 1000 words, 0.2% of the GPU’s 10.12-second synthesis time.
- The CUDA build took ~46 minutes on this 16-thread machine for
sm_86; the CPU build is effectively instant by comparison.
What was measured ¶
Kokoro-82M is small enough to run on a CPU, which is why it is a nice local text-to-speech model. It is also small enough that a GPU does not automatically make a one-shot command faster: the fixed cost of creating an ONNX Runtime session and a CUDA context can eat the gain.
This benchmark separates those two effects:
- Warm synthesis loads the model once and then times repeated
Kokoro.create()calls. This is what a long-lived process pays. - Cold latency starts a fresh
kokoro-speakprocess for every measurement, including Python startup, model load, CUDA setup, synthesis and WAV write. This is what a user pays for a one-shot CLI invocation.
All synthesis uses the af_heart voice at speed 1.0, 24 kHz mono output. The text is generated by repeating a natural paragraph and trimming it to exactly the requested number of words. Warm timings include espeak phonemization, tokenization, chunking and all model inference. Cold timings additionally include process start and model load.
The software stack is onnxruntime 1.27.1 and kokoro-onnx 0.6.1, packaged through Nix. The CPU build uses the same onnxruntime version without CUDA. The GPU build uses CUDAExecutionProvider with CPUExecutionProvider as fallback.
Hardware ¶
| Part | Model |
|---|---|
| CPU | Intel Core i7-11700, 8 cores / 16 threads, 2.5 GHz base, 4.9 GHz boost |
| GPU | NVIDIA GeForce RTX 3060 Laptop GPU, 6 GB, 80 W |
| GPU driver | 595.112.07 |
| CUDA / cuDNN | CUDA 13.4 / cuDNN 9.27 |
| OS | NixOS 26.11, Linux 6.18 |
One caveat: the machine was bought as an RTX 2080, but the driver reports an RTX 3060 Laptop. The benchmark is for the GPU the driver actually exposes. If the card is a desktop RTX 2080, expect roughly 1.2–1.5× the GPU throughput measured here; if it is an 80 W laptop 3060, the table is the measurement. Either way, the CPU numbers are the same.
Method ¶
Warm runs used the kernel’s performance CPU governor, restored to powersave afterwards. Without pinning the governor, the CPU baseline varied by 20–40% between runs because the clock ramps with load. Cold runs used the normal powersave policy, since that is what a user actually experiences.
The reported warm numbers are medians of five runs up to 100 words and three runs for 300–1000 words. The cold numbers are medians of five fresh processes.
xRT means audio seconds produced per wall-clock second. RTF is its inverse. Higher is better.
Warm synthesis ¶
| Words | Audio | CPU synth | GPU synth | Speed-up | CPU xRT | GPU xRT |
|---|---|---|---|---|---|---|
| 5 | 1.7 s | 0.40 s | 0.08 s | 5.08× | 4.20× | 21.24× |
| 20 | 8.0 s | 1.90 s | 0.26 s | 7.32× | 4.20× | 30.67× |
| 50 | 17.3 s | 4.56 s | 0.52 s | 8.72× | 3.79× | 33.08× |
| 100 | 35.1 s | 8.97 s | 1.13 s | 7.97× | 3.91× | 31.22× |
| 300 | 99.0 s | 30.58 s | 3.03 s | 10.09× | 3.24× | 32.64× |
| 1000 | 328.0 s | 102.59 s | 10.12 s | 10.14× | 3.20× | 32.41× |
Geometric mean: 8.01× faster on the GPU.
The same numbers as throughput per word:
| Words | CPU ms/word | GPU ms/word |
|---|---|---|
| 5 | 80 ms | 16 ms |
| 20 | 95 ms | 13 ms |
| 50 | 91 ms | 10.4 ms |
| 100 | 90 ms | 11.3 ms |
| 300 | 102 ms | 10.1 ms |
| 1000 | 103 ms | 10.1 ms |
The GPU is not just “a bit faster at the model”. Once the sentence is long enough to amortize kernel launches, it produces about ten times as much audio per second. A five-minute report drops from about 103 seconds of CPU synthesis to about 10 seconds on the GPU.
Cold command latency ¶
| Text | Audio | CPU cold | GPU cold | GPU vs CPU |
|---|---|---|---|---|
| 7-word sentence | 1.9 s | 1.43 s | 1.66 s | 0.86× |
| 57-word paragraph | 19.3 s | 5.79 s | 2.08 s | 2.79× |
A fresh CUDA session costs about 1.5–2.4 seconds against 0.6 seconds for the CPU build. That is why the short sentence loses: the GPU has less than two seconds of audio to work with, and the setup cost dominates. The crossover is around 20–30 words.
The cold paragraph is only 9.3× realtime, against ~32× warm, because the process pays model load and CUDA setup again. If you are building a daemon or a long-lived agent process, the warm numbers are the relevant ones. If you run echo "..." | kokoro-speak once, the cold numbers are.
Where the time goes ¶
Three things stood out:
- The model is the bottleneck, not espeak. Phonemization takes 0.02 seconds for 1000 words: 0.2% of the GPU’s 10.12 seconds. I expected text processing to cap the GPU speed-up and it does not.
- CUDA startup is a fixed tax. The warm model load is 0.61 seconds on CPU and 2.37 seconds on GPU. Every fresh CLI invocation pays it again.
- The CPU degrades with length; the GPU does not. CPU throughput falls from 4.20× realtime at 20 words to 3.20× at 1000 words. The GPU stays between 30.7× and 33.1× realtime once there is enough work to fill it.
What about an RTX 2080? ¶
If the GPU in this machine really is the desktop RTX 2080 rather than the 3060 Laptop the driver reports, the measured table is your 2080 table. If it is the 3060 Laptop and you have a separate desktop 2080, the 2080 has 2944 CUDA cores, 184 tensor cores, 10.1 TFLOPS FP32, 448 GB/s of memory bandwidth and a 215 W budget. For a small fp32 ONNX model, that should be roughly 1.2–1.5× the CUDA throughput measured here.
That would put the warm numbers at approximately:
| Words | Speed-up over CPU | GPU xRT |
|---|---|---|
| 5 | 6.1–7.6× | 26–32× |
| 1000 | 12–15× | 39–49× |
The cold-start penalty stays in the same 1.5–2.5 second range, so a one-sentence command is still roughly parity with the CPU; a paragraph cold run should be about 3–3.5× faster.
Reproduce ¶
The benchmark lives in the kokoro-speak skill as bench/bench.py, bench/cold.py and bench/compare.py. The CUDA package is selected with cudaSupport = true.
nix run .#kokoro-bench-cpu -- --label cpu --json-out /tmp/cpu.json
nix run .#kokoro-bench-cuda -- --label cuda --json-out /tmp/cuda.json
nix shell nixpkgs#python3 -c python3 bench/compare.py /tmp/cpu.json /tmp/cuda.json
The full run is in BENCHMARK.md in the skill directory. The short version: for a one-shot command, use the CPU for very short replies; use the GPU for paragraphs, reports and anything that synthesizes more than roughly 20–30 words. For a long-lived process, the GPU wins almost everywhere.