PushToType can run a lot of speech engines now. Austin wanted to know which one should be the default, so we measured them.
The test was small, and we know it
Eleven engines ran through the app's own worker code on an M1 Max with 64 GB. The test set was 45 clips, 326 seconds of audio in total.
24 clips were developer-jargon sentences read by two macOS text-to-speech voices, with terms like kubectl, git rebase, useEffect and Postgres. 20 were short LibriSpeech clips of one person reading. The last was a two-minute read that stands in for a long push-to-talk take.
The Mac was shared with other jobs, and the load average ran from 4.5 to 25. That makes the latency numbers upper bounds. Treat accuracy gaps under about a point as noise.
Whisper Small.en is the default we'd pick
It caught 31 of 36 jargon terms before any dictionary fixes, more than any other engine. Its word error rate on read speech was 3.2%, and its median latency was 0.72 s.
It also holds no memory between takes. whisper.cpp starts fresh for each one, uses about 0.7 GB while it works, and needs no Python.
Parakeet v2 is the most accurate until you talk for two minutes
Parakeet v2 had the lowest error on read speech: 2.3% on the short clips and 2.9% on the two-minute take.
The problem is that it decodes the whole take in one pass. On the two-minute clip its memory footprint hit 47.6 GB. On an 8 or 16 GB Mac, that take would swap hard or get killed. It needs chunking before it's safe as a default.
Parakeet v3 has the same memory problem, and it returned an empty transcript for one 1.9-second clip.
Phonon 2 is the fastest, and it costs you 3 GB
Phonon 2 had a 0.11 s median latency and finished the two-minute take in 1.85 s. It was good on read speech too, at 2.6%.
The catch is memory. It keeps about 3.1 GB resident while idle, takes 13.5 s to start, and needs a 1 GB Python environment. It's also weaker on jargon before corrections, with 21 of 36 terms.
Full disclosure: Phonon 2 is the default engine in Super Notch, which we build.
The Bark models struggle with jargon
All three Bark models were fast. They were also the weakest on jargon, getting 16 to 18 of 36 terms before corrections. Each one writes a 1.2 GB cache on first load and peaks at 7 to 9.5 GB while decoding. Distil ties Bark Plus at 18 raw jargon terms.
Two engines didn't get measured
macOS Speech, PushToType's current first-run engine, needs a person to click Allow on the speech-recognition prompt. The app also streams it from the live microphone, which a file-based test can't drive without code changes. It stays the first-run engine for now because it needs no download.
OpenAI's gpt-4o-transcribe is paid, needs your own API key and sends audio off your Mac. Nobody signed off on the key or the spend for this run, so we skipped it.
The dictionary helps, and now we know its gaps
PushToType runs a corrections pass after transcription. With it on, Whisper Small.en and both Parakeet models reached 32 of 36 jargon terms.
Some misses never get fixed. "use effect" stays two words, "OpenAI API" comes out as "Open IAPI", and Postgres turns into "postgars". Those are the next dictionary entries.
What else broke
Whisper Small, Base and Tiny each dropped about 58 of the 269 words in the two-minute take. Forcing English didn't change it. The likely cause is the no-speech settings skipping a 30-second window.
Distil-Whisper got stuck repeating one phrase on the long take.
Read this with the caveats
The human audio is one person reading prepared text, the jargon came from two synthetic voices, and it's 45 clips on a busy machine. That's enough to pick a default. It isn't the final word.
The full tables, model hashes and method are going into docs/benchmarks.md in the PushToType repo with the next PR.