Bench · Speech to text · October 2, 2026
Which voice model hears you?
Holly listens on your own devices. So we timed and scored the speech recognizers it can use, on English, Chinese, and both in one sentence.
Speed and accuracy
Lower left is better: fewer errors, less waiting. Each dot is one recognizer, over turns of real speech.
Accuracy
Errors per character, against what the speaker checked by ear. Lower is better. Every recognizer on the Mac heard the same turns.
*Apple's recognizer hears one language at a time. Buddy runs one for each of the 2 languages you set (here English and Chinese) together, and keeps whichever wrote more words in its own language. That's the row scored here.
†For comparison: a cloud model (GPT-5.6) asked for a word-for-word transcript of each whole recording. Not timed, and it has a head start: for English and mixed, the answer key began as its own earlier transcript, which the speaker then checked.
Speed
Time from the end of a turn to its words, on a MacBook Air. A typical turn was seconds long; the longest, .
*Here Apple was handed each finished turn at once, and the 2 languages count as the slower of the two. On a call it listens while you talk, so its wait after you stop should be shorter. We haven't measured that yet.
The models
How big each one is, and the exact copy we ran. Bigger and more precise usually means slower and more accurate.
Parameters are the model's learned numbers, as OpenAI publishes them. Precision is read from each model file. Palettized means each layer's numbers are snapped to a small shared set (16 values for 4-bit, 64 for 6-bit), which makes the file smaller: this turbo is 1.0 GB, where a 16-bit copy would be about 1.6 GB. The turbo we tested is that compressed copy, so some of its errors may come from compression; the full 16-bit turbo is untested. Apple doesn't publish its recognizer's size.
Listen and compare
Play a recording, then see exactly what each recognizer wrote, turn by turn, as it wrote it. The recordings here are turned up for listening; the recognizers heard the originals.
What we learned
- Mixed speech is where they split apart. On English, every Whisper model scored . On English and Chinese in one sentence, .
- The biggest model was the most accurate, and about half a second slower per turn than turbo.
- Short turns can fool language detection. Whisper picks one language per turn. Twice it picked wrong: it wrote a Chinese sentence in Japanese, and kept only the English opening of a mixed one.
- Each Apple recognizer garbles the other language. The English one heard 我旁边有窗户 as "Towards higher". Keeping the better of two works when a turn is one language, and loses words when it's both.
- Forcing one language makes it worse. Told to hear Chinese, Whisper rewrote English sentences as rough Chinese instead of writing them down.
- Results move between runs. The same model on the same mixed turns scored 5% one run and 8% the next. Small differences in these tables aren't meaningful.
A bug we found
Calls ask Whisper to work fast: no timestamps, a cap on words. Past 30 seconds Whisper needs timestamps to find where to keep reading, so a long turn lost most of its words. On a 65 second recording the biggest model kept 1 of 65 English words. The fix keeps timestamps on for anything over 30 seconds, and the same recording now comes through whole.
How we tested
- The Mac. .
- Turns. Each recording was cut wherever the speaker paused for 0.7 seconds, the way a call ends a turn: turns in all.
- Whisper ran through WhisperKit (Core ML) on the Mac's Neural Engine, loaded and warmed up first, with the same fast settings Teamforce uses on calls. Each turn ran twice; the second run counts.
- Apple's recognizer ran on the Mac, on device only, set up the way Buddy sets it up on the iPhone. Same kind of model, maybe not the same version.
- The answer key is what the speaker said, checked by ear while listening. For English and mixed, it started as a cloud model's transcript that the speaker confirmed word for word; for Chinese, the speaker wrote it.
- Scoring. Errors per character: the edits needed to turn what a recognizer wrote into what was said, divided by its length. Before counting, both sides were made lowercase and simplified Chinese, without spaces or punctuation, with numbers written one way (三 and 3 are the same), and 他, 她 and 它 counted as one, since they sound the same.
What this test can't tell you
- It's one speaker and under three minutes of speech. Your accent, your room and your microphone will move these numbers.
- It's one Mac. A Mac with a bigger chip may be faster; we haven't measured one yet.
- The answer key isn't perfect either. In three spots in the Chinese recording, most recognizers heard something the speaker's transcript doesn't have (想要 for 要, 所以 at the start of a sentence, 翻译 for 发音). We kept the speaker's version.