How speech-to-text works, in plain English

Speech-to-text is software that turns spoken words into written ones. The field is also called automatic speech recognition, or ASR. Wikipedia describes it as methods and technologies that "translate spoken language into text or other interpretable forms"¹.

You'll also see the word transcription. In a business setting it usually means the same thing: audio goes in, a written transcript comes out.

The basic steps

A transcription tool doesn't hear the way you do. It works on numbers that describe the sound. Then it guesses.

First the audio is cut into very short slices. Then software compares each slice with patterns it has learned and works out which speech sounds are most likely. Last, it joins those sounds into words and sentences.

Two kinds of knowledge

Older systems split the job into two models. An acoustic model connects sound to speech sounds. A language model judges which word sequences are likely in a language. Wikipedia says both "are important parts of statistically based speech recognition algorithms"¹.

The language model is why a tool can pick "weather" over "whether" from the words around it. It's also why odd names get mangled. A name the model rarely meets has a low chance of being picked.

Newer "end-to-end" models "learn from all the components at once"¹. Which design a tool uses is the maker's choice.

What a transcript contains

Plain text isn't all you get. Amazon's documentation says a transcript from its service has "confidence scores and timestamps for each word or punctuation mark"². A confidence score is the tool's own estimate of how sure it is about a word. It isn't a guarantee that the word is right.

Not every tool shows these extras. Check the maker's documentation for what your tool returns. The page on timestamps and speaker labels explains the time marks in more detail.

Machine learning, not magic

Amazon's documentation says its service "uses machine learning models to convert speech to text"². A machine learning model is software that learned patterns from many examples instead of following hand-written rules.

Two things follow. The tool does best on speech that resembles the examples it learned from, which is why a strong accent or an unusual word can trip it up. And it can be wrong without any warning at all.

How accuracy is scored

Accuracy is "usually rated with word error rate (WER)"¹. Word error rate counts the words a tool got wrong compared with a correct reference text. A lower rate means fewer mistakes.

Makers sometimes publish such figures. Treat them with care. The score depends on the test audio, and that audio may be cleaner than a real recording of your Tuesday team meeting. For the causes of errors, read why transcripts get words wrong.