Real-time AI video dubbing looks like magic. You press play, and within a fraction of a second the audio is replaced with a different language. Behind that, three AI models run in a chain. Here's each step.
Step 1 — Speech-to-text
The first model listens to the source audio and turns it into text. Modern speech-to-text models (Whisper, Canary, USM) can do this in under 100ms for short clips. The trick is that they have to be streaming — they don't get to wait for the speaker to finish a sentence. SyncDub uses a streaming variant that emits partial transcripts every 80 milliseconds.
Step 2 — Translation
The second model takes the partial transcript and translates it. We use a small multilingual model with a 4-sentence context window — enough to disambiguate idioms and gendered references, not so much that latency suffers. Translation runs in about 80–120 ms.
Step 3 — Text-to-speech
The third model turns the translated text into audio. This is the slowest step — typically 150–250 ms — because generating natural-sounding speech is computationally expensive. We use a single voice across all 70+ languages, which is faster (no per-language voice cloning) and sounds more consistent.
What we do with the time
The three steps add up to 320–450 ms in practice. The total budget — from the speaker saying something to the dub being heard — is around 320 ms for VOD and 1500 ms for live streams. Live streams have a higher budget because we can't predict what's about to be said.
What's hard about this
- Background noise. Music, laughter, traffic — all of it gets fed into the model as if it were speech, and the model has to figure out what's a word and what's not.
- Multiple speakers. Overlapping dialogue is hard for any speech-to-text model.
- Code-switching. People mix languages in the same sentence — 'So I was at the café and he was like, ugh, this ismeta.' Real speech is messy.