Streaming Transcription

Streaming transcription refers to the conversion of spoken words into written text while the speaking is still happening. The opposite approach would be to wait until someone has finished speaking, then process the recording in full, and only afterward deliver the result. With streaming, audio flows in small chunks — often just a few tenths of a second long — continuously into a speech recognition system. The system returns what it has understood so far and constantly updates this text. The user watches the words appear, almost like typing.

Why milliseconds make the difference

Whether a system feels natural depends heavily on its delay — experts call this latency. In a phone call with a voice assistant, you don’t want to wait three seconds for a reply after you’ve finished speaking. That feels like a bad international call with a stuttering connection. Streaming transcription keeps this delay low because the system doesn’t wait for the end of the sentence but starts working immediately.

For people with hearing impairments, low latency is even more important. Live captions at an event or in a video conference are only useful if they stay close on the heels of what’s being said. Captions that appear five seconds late lose their purpose. The same applies to real-time translators that are supposed to immediately generate text in one language from spoken words in another.

How the system speaks before it has finished listening

The speech recognition system receives the audio as a continuous data stream, known as the audio stream. It breaks this stream down into short windows of about 20 to 30 milliseconds and analyzes each window individually. Based on the preceding windows, the system calculates probable words — and outputs a preliminary hypothesis. This hypothesis can still change: if the system hears another word that steers the sentence in a different direction, it sometimes retroactively corrects the text it has already produced. This is occasionally visible as a brief flicker in live captions.

Modern systems use neural networks for this purpose — models that have learned from many millions of examples of human speech which sound sequences correspond to which words. An architecture called Transformer has proven especially effective here, as it is good at capturing the relationship between words that are far apart from each other. The catch with streaming is that a Transformer actually wants to see the complete sentence before making a judgment. Special variants, so-called streaming transformers, therefore work with an artificially limited field of view ahead — they only “look” a small distance into the future of the audio, which keeps latency low.

On top of that comes the detection of sentence boundaries in real time. When a sentence ends and a new one begins is not marked as clearly in spoken language as it is in written text. The system must simultaneously evaluate pauses, intonation, and typical sentence lengths in order to punctuate sensibly.

Where streaming transcription shows up today

The best-known application is automatic captioning in video conferencing programs like Google Meet or Microsoft Teams. Both services transcribe what is said live and display it at the edge of the screen. Voice assistants like Siri or Google Assistant also rely on streaming transcription: they begin to understand the request while you are still speaking, which makes them appear responsive.

In journalism and in parliaments, streaming transcription is used to make speeches immediately available as text transcripts. Broadcasters use it for real-time captions on television, which is legally required in many countries. And medical dictation systems transcribe a doctor’s report into structured text directly as it is spoken — this saves writing work and speeds up documentation.

The topic comes up in public discussion whenever new models like OpenAI’s Whisper or services like AWS Transcribe Streaming announce new accuracy records. This almost always involves the same trade-off: better recognition accuracy on one side, lower latency on the other. Maximizing both at the same time remains one of the central challenges of speech processing.