When we measured our first end-to-end dub in May 2026 the round-trip was 1.4 seconds. The model itself was 380ms. The rest was everything around it: websocket frames, audio buffers, the manifest fetch, the offscreen handoff, the way we were chunking 16 kHz PCM into 80 ms windows. Two months later we're at 320ms. The model is the same.
What was actually slow
Three things, in this order of damage: (1) we were decoding the source video audio twice, once in the content script and once again in the offscreen document; (2) we were sending the entire 80ms PCM window as a single websocket frame, then waiting for a full response before scheduling playback; (3) every playback chunk triggered a re-fetch of the user's voice preference, which round-tripped through Chrome's storage.
What we changed
- Decoded once. The content script owns the AudioContext and ships 16 kHz PCM into a shared worker; the offscreen document reads from a ring buffer, not from the video element.
- Stopped waiting for full responses. The model streams partial tokens. We start playing back as soon as we have ~120ms of decoded audio in the buffer, and we backfill the rest.
- Cached voice preferences in memory. The offscreen document loads them once at startup and never goes back to chrome.storage mid-session.
What we didn't change
The model. People keep asking whether we swapped to a smaller, faster model. We didn't. We replaced the slowest things around the model and discovered, embarrassingly, that we had almost a full second of slack that was just plumbing.
"Latency lives in the joins. The model is rarely the bottleneck."
What's next
We're aiming for 220ms. The remaining wins are smaller — likely another 30–40ms from speculative decoding on the next token and 50–60ms from collapsing two websocket frames into one. We'll publish the numbers when we get there.