Text to speech (TTS) technology transforms written words into spoken audio. The system first works out what the text says and how it should sound, then an AI model generates the audio.
Initially, a TTS engine normalizes text input and converts it into mathematical tokens and vector representations that a computer system can manipulate. Next, the model transforms those mathematical constructs into digital representations of sound and finally into audio output.
In this article, we’ll explore how different text to speech tooling works and how the latest generation of generative AI neural speech synthesis offers the most natural speech yet.
Summary
- Text to speech systems convert written words into audio in two stages. They convert text into tokens, or mathematical representations of language. They then convert the tokens into an acoustic representation that’s output as sound.
- Newer neural TTS engines use AI models to produce natural-sounding speech. Business teams use them for content creation, subtitling, and powering customer support agents.
- Pronunciation dictionaries and phonetic tagging are two ways to fine-tune TTS output to sound as natural as possible and ensure technical jargon and names are pronounced correctly.
- A Text to Speech API allows you to connect business applications to high-quality voice generation.
What is TTS? How text to speech technology works
Text to speech works by analyzing written text, deciding how each word should sound, and then using both a neural model and a vocoder to generate the spoken audio.
TTS broadly breaks down into two phases:
- Frontend: The system converts text into a mathematical form the TTS engine can work with.
- Backend: The system takes the output of the first phase and transforms it into waveforms modeling speech.
Let’s break down the components of how TTS works in more detail.
Text normalization and linguistic analysis
The first step in a TTS process is natural language processing (NLP): cleaning, tokenizing, and disambiguating raw text input. It breaks text into tokens, individual units of written language that a TTS engine can manipulate. A token might be an entire word, a “subword” unit, or punctuation.
Text often contains symbols other than full words, like abbreviations and numbers. A TTS engine also needs to determine how to speak those items. For example, the TTS engine reads “$11” and converts that text into the sounds for “eleven dollars.”
Semantic meaning and prosody
Prosody is the pitch and rhythm of speech, which is what makes TTS sound natural. While processing text, the engine also generates a layer of prosodic information that maps changes in stress and intonation onto text to make it sound more natural.
A TTS engine also needs to adapt its text to speech pronunciation based on context. It will assemble this information in the form of vector representations, numerical data that maps the semantic similarity between words and phrases. Think of it like an X-Y graph where a line is plotted along a number of different points, but mapped in many more than two dimensions.
One specific example is homographs. Homographs, or different words that share a spelling, sometimes need to be pronounced differently. The TTS engine needs to identify the intonation and stress to use on each when speaking. For example, how to pronounce the grapheme “desert” in the sentence "The soldier wanted to desert his unit by hiding in the desert.”
Watch Eleven v4 bring this example to life.
The soldier wanted to desert his unit by hiding in the desert.
0:00
Acoustic modeling and waveform (speech) synthesis
The TTS engine takes the tokenized information and contextual vectors, all of which are now numerical data, and converts them into an acoustic representation, such as a mel-spectrogram. This includes different frequencies, energy levels, and sounds that the brain interprets as emotion. Many different systems have been used to perform this function in the past, but today neural models are among the most effective.
The final step uses a vocoder to convert these spectrograms into audio that human listeners recognize as speech. This process is called waveform generation.
How TTS works in action: A worked example
The easiest way to see how the stages of TTS fit together is to follow a single sentence through the pipeline.
Take the text “Mr. Ríos paid $11 on 10/06.”
- Normalization: The engine expands “Mr.” to “Mister,” reads “$11” as “eleven dollars,” and decides whether “10/06” is “October sixth” or “the tenth of June,” depending on the locale and context.
- Tokenization and context: The cleaned text is split into tokens and converted into vectors that capture how the words relate to each other.
- Prosody: The engine places stress on “eleven dollars,” adds a short pause before the date, and lets the pitch fall at the end of the sentence.
- Acoustic model: The neural model turns the tokens and prosody plan into a mel-spectrogram that describes the sound over time.
- Vocoder: The vocoder converts the spectrogram into the final audio waveform you hear.
Each stage hands a more detailed plan to the next, which is why errors early in the pipeline, such as a misread date, carry through to the finished audio.
Why does neural text to speech sound more human?
Neural speech synthesis excels at producing natural, intelligible speech, which are the two qualities engineers use to evaluate TTS engines. Neural TTS works by training a deep neural network (DNN) to produce speech from a text input. You train the DNN on large volumes of recorded speech paired with the associated text.
Neural speech synthesis engines are only the latest iteration of enterprise-grade TTS. Two prior generations of commercial synthesis technology preceded them.
Formant synthesis was the dominant mode in the 1970s and ‘80s. Instead of sampling human speech for runtime output, it generated synthetic sounds that modeled speech. It was intelligible, but monotone and robotic-sounding.
Concatenative synthesis dominated from the 1990s through the 2010s and sounded more natural than earlier formant systems. Instead of generating tones based on mathematical rules, they stitched together, or concatenated, small human speech samples. These systems used linguistic lookup tables to match normalized text to sounds, then output small units of concatenated speech audio pulled from a massive library, or “corpus.” While more natural-sounding than speech from a formant engine, concatenation can still make speech sound choppy or cut up.
What’s the difference between an AI voice generator vs. text to speech?
“AI voice generator” is a term often used in marketing to describe a broad range of voice technologies, including text to speech. Text to speech refers specifically to systems that take written text as input, model its semantics and prosody, and output audio waveforms.
AI voice generators are a full ecosystem of interrelated voice technologies, including TTS and other generative systems for voice design, voice cloning, and voice style transfer. While it is often used as an interchangeable term for text to speech in marketing, AI voice generators are best considered a term for the broader category of generative voice tooling of which TTS is a part.
How to use text to speech for content, accessibility, and support
Text to Speech models have become a central part of workflows around the world, both within personal and commercial fields.
Here are some of the leading use cases of text to speech.
Content creation
Text to speech tooling can help reduce the costs of large-scale media production and reduce time spent in the studio for narration and post-production pickups.
Example use cases include:
- Narration
- Voiceovers
- Dubbing
- Audiobook production
- Game dialogue
For example, TAP used ElevenLabs TTS to scale educational content for schools across India.
Accessibility
Text to speech is one of the most widely used assistive technologies because it lets people take in written content through listening. Natural speech improves listener engagement and reduces mental fatigue when listening to long-form material.
Common TTS accessibility uses include:
- Screen readers: Read entire pages aloud with a screen reader and TTS.
- Dyslexia and literacy support: Reduce the burden of reading by accompanying reading sessions with a TTS voice.
- Low vision and blindness: Screen readers speak interface elements and page content aloud, so people can navigate software and websites without seeing the screen.
- Cognitive and attention differences: Hearing text can lower the effort of reading long documents and make it easier to stay focused.
- Learning and language study: Students hear correct pronunciation of new vocabulary and can listen to textbooks while commuting or studying.
Screen readers are full accessibility tools that describe everything on a screen, including buttons and menus. Read-aloud tools focus on speaking the content itself, usually with highlighting that follows along, and they benefit from more natural voices because people use them for long stretches.
Customer support
Neural speech synthesis can provide a high-quality customer experience when paired with an LLM capable of agentic interactions. Platforms such as ElevenAgents combine the voice layer with speech recognition, an LLM, and connected tools that let an agent understand what a caller needs and respond in a natural voice.
Agents built with ElevenAgents work across phone, web, WhatsApp, email, and SMS to allow customers to continue a conversation across channels without repeating themselves. Example use cases include:
- Bidirectional voice AI agents: Handle live conversations by answering questions froma knowledge base or routing customers into specific flows, such as refunds or purchases.
- Interactive voice response (IVR) systems: Replace outdated ‘Press 3 for sales’ menu trees with conversations where your customers can speak naturally.
- Multilingual support: Serve customers in their own language from a single agent setup.
Klarna, for example, reduced time to resolution by 10x with ElevenAgents
How do text to speech engines control pronunciation?
Modern neural speech synthesis engines can generate very natural-sounding speech, but edge cases will always exist in something as broad, deep, and dynamic as human language. You may want to keep accents, slang, or regional pronunciations in your generated speech. You might be producing audio in an industry that has unique pronunciations for technical terms. You might simply want your brand name pronounced in a very specific, consistent way.
Pronunciation dictionaries and phoneme overrides
ElevenLabs’ ElevenAPI includes a pronunciation dictionary that allows you to customize how the engine speaks. You build a dictionary in a W3C-standard Pronunciation Lexicon Specification (PLS) file that maps graphemes (symbols in text) to phonemes (spoken words broken down into sounds). When the TTS engine encounters one of the listed graphemes, it overrides its trained pronunciation and uses the phonetic pronunciation from the dictionary.
You can learn how to build a pronunciation dictionary in ElevenAPI in the Using pronunciation dictionaries documentation.
Native IPA and phonetic prompting
Some TTS models can also recognize in-line pronunciation tags embedded directly in text input. So instead of building an account-wide dictionary, if you just want to tweak pronunciations in one or two audio clips, you can tag the words in text using standard IPA notation. For example, if you’re recording an audiobook and want a fictional character’s name pronounced a specific way.
Common text to speech challenges and how to address them
Even the strongest TTS models can struggle with the nuances of human language. Two of the most common issues are mispronouncing novel words and delivering monotone speech. Here’s how to address them.
Mispronunciations of unfamiliar words, jargon, and proper nouns
TTS engines use algorithms to predict how normalized text should be pronounced based on how graphemes map to phonemes in their training data. They take their best guess based on the text and context they’ve ingested. But as we discussed in the previous section, there are going to be many edge cases where you want something pronounced a very specific way.
Pronunciation dictionaries or targeted IPA tags are going to be your best tools for these edge cases. If you have a commonly recurring term you want pronounced consistently, a dictionary is the best solution. If you just have targeted tweaks you want to make to a few terms, use phonetic IPA tags.
Monotone delivery
These issues are most common when processing longer scripts when the same words or phrases are spoken over and over. An engine’s synthesized speech may sound great in 10-second soundbites, but hearing the same rote pronunciation and rhythmic patterns over a 45-minute lecture quickly becomes fatiguing to the listener.
Vary the phrasing used in your scripts and make judicious use of phonetic tags to periodically vary pronunciations. Use emotional and rhythmic cues to make the speech output more dynamic, rising and falling with different narrative beats.
Build natural voice experiences with a Text to Speech API
Lightweight web tools can be great for producing small audio clips, but they don’t scale well to larger business needs. If you’re trying to manage thousands of customer interactions per minute or generate high-quality, long-form audio content, you’re going to need more robust TTS tooling.
A Text to Speech API like ElevenAPI allows you to connect your software directly to neural speech synthesis models. You can use REST requests or persistent WebSockets to generate audio on demand for agents or other purposes at scale.
How a text to speech API request works
Although the models are complex, the developer flow is easy to understand and get started with.
Using ElevenAPI, you can integrate Text to Speech into your applications in minutes:
- Send a request: Your application sends the text, a voice ID, and a model ID to the API with your API key.
- Generate: The model runs the pipeline described above, from normalization through to the vocoder.
- Receive audio: The API returns the finished audio file, or streams it back in chunks so playback can begin before generation finishes.
- Play or store: Your application plays the audio live, saves it, or passes it on to another system such as a phone line.
The same flow applies whether you use REST for finished files or WebSockets for live streaming, and the SDKs described below wrap these calls for you.
Architectural considerations for developers
Depending on your use case, you may need to balance trade-offs between processing speed, audio fidelity, network latency and protocols, and integration capabilities.
Latency vs. fidelity
For conversational agents to sound natural, you want to aim for approximately 100ms time-to-first audio, which means minimizing latency across the network and to the TTS engine. The consideration here is the quality of the audio output you’re able to get from your agent’s architecture within that constraint.
Conversely, latency matters less when you’re producing long-form audio, like a podcast or lecture. Rather than minimizing latency, you’ll want to maximize high-fidelity output.
Streaming protocols
The protocols you use for your API connection can also impact latency and performance. WebSockets allow for true bidirectional audio streaming, meaning you can process chunks of text at a time rather than wait for an entire feed to render. Playback can start on the first chunks while the TTS engine still processes later ones.
SDKs
Enterprise-grade Text to Speech APIs, like ElevenAPI, offer full software development kits (SDKs) for Python and TypeScript/JavaScript, as well as REST APIs, to help you stand up new architecture fast. Use them to reduce integration overhead for your dev team.
Get started with ElevenAPI for Text to Speech
Natural TTS synthesis from a leading speech model produces the high-quality audio output your agents and audio productions need to keep listeners engaged. Whether you’re looking for a high-quality conversational agent to support customer workflows or want to build TTS into your application, ElevenLabs has a solution.
Get started today by producing your own high-fidelity audio with ElevenAPI, or learn more about ElevenAgents.
.webp)
