babel

Picking a translation engine on published numbers is impossible, because every model reports its latency on different hardware, different audio and a different definition of when the clock stops. The only way to choose is to run them all in one condition and time them the same way.

So the harness measures four things per engine: time until the listener has anything at all, which is what governs how the conversation feels; time to finish the whole utterance; processing time as a fraction of audio duration; and translation quality against reference translations.

The corpus is deliberately real recorded speech rather than synthesised audio. Synthetic speech is too clean, and a system tuned on it will not survive a room.

Doing speech-to-English in one step collapses on Chinese. It does not degrade — it fails, scoring under 1 BLEU, and it is slower while failing, because a model that cannot translate keeps generating. Routing through a dedicated translation stage instead lifts it roughly eightfold and runs several times faster.

The same change is not a general win, which is the more useful finding. On Spanish the two paths are level, and on this Japanese set the cascade is clearly worse than going direct. An architecture decision that looked like "always cascade outside Europe" is really per-language, and no amount of reading model cards would have said so.

The harness ships with a written list of what it does not yet cover, which is the part most benchmarks leave out. The big one: everything here measures turn latency, where the clock starts when the speaker stops. Real systems emit while you are still talking, which changes the metric entirely and is not implemented yet.