VOD and live translation look like the same problem until you actually try the second one. The first version of our live mode hit 6 seconds end-to-end — fine for a video nobody's watching in real time, totally unusable for a Twitch stream with 4,000 people in chat.
What broke first
We assumed we'd reuse the VOD pipeline. We were wrong. VOD lets you look ahead — the next 5 seconds of audio is already decoded, and the model can plan ahead. Live audio is the opposite: every millisecond you look forward is a millisecond your viewer hasn't heard yet. We had to redesign the model wrapper to handle unbounded streams, not 30-second clips.
What we did differently
- Switched to a streaming STT model that emits partial transcripts. The translation model gets the partial transcript, not the finalized one.
- Built a fallback queue: if the translation model lags more than 800ms behind the source audio, we play a 'waiting' tone and skip ahead, rather than letting audio pile up.
- Added a 'live mode' opt-in to the extension so users can choose between latency and quality for each site.
Where we landed
End-to-end is now under 1.5 seconds on most streams. The visual delay between the source speaker's lips and the dubbed audio is noticeable but acceptable. We're aiming for 1 second by Q4 2026.
"Live translation isn't faster VOD. It's a different product."