A native word-level transcription model takes an audio recording or real-time stream as input and outputs a structured array of tagged and timestamped token objects representing speech and non-speech sounds. Each token is one of three types: word, spacing, or audio_event. Audio events can be laughter, applause, or other background sounds.
In this article, we'll explore how audio transcription with timestamps works and why it's a more flexible and powerful form of transcription than conventional transcription that relies on forced alignment.
Summary
- Word-level transcription directly outputs structured, timestamped arrays of speech data and non-verbal audio events.
- This contrasts with conventional automatic speech recognition models, which output full blocks of text. To timestamp that to raw audio, you need to perform a second forced-alignment process, which takes longer and adds compute overhead.
- Event tags help you capture non-verbal content, like laughter or applause. Because this data is searchable, you can use it to help identify highlights or viral moments based on audience reactions.
- You can feed the structured output data to applications for a number of use cases, including highly accurate captioning, searchable audio archives, and automated social clipping.
- Scribe’s word-level transcription can support up to 5 channels, each independently transcribed.
What is audio transcription with timestamps and event-tags?
Timestamped transcription is a form of automatic speech recognition (ASR) that tracks the precise start and stop points of words and other audio events, like applause, as it transcribes. Its output is fully structured, navigable data.
Conventional ASR transcription parses audio into large blocks of human-readable text. It gives you captions and a transcript for human readers, but it’s not very useful as data.
If you want to structure it with timestamps, you need to run it back through a secondary machine learning service to map the text to the audio. You could eventually generate similar output, but it takes multiple processing steps rather than coming natively from the API, and so it adds some latency and extra compute overhead.
Adding navigation to audio
The key advantage of timestamped, event-tagged transcription is that it is both human- and machine-readable. This has some important business applications:
- Compliance auditing: You can run a search on a transcription for keywords and get exact timestamps for any mention of sensitive data.
- Video production and editing: Video production software can synchronize subtitles to the millisecond with what's being said on screen. It can also label audience reactions appropriately.
- Searchable archives: Public relations, sales, and training teams, among others, can transcribe large volumes of customer interviews, trade show speeches, webinars, or town halls into massive, searchable archives.
- Marketing analytics and customer sentiment: Event-tagging can label audience reactions that might vanish from basic text transcription.
What are the advantages of getting native word-level timestamps?
Standard speech to text models process speech in chunks, usually whole sentences or paragraphs. If you want word-for-word timestamping, you need to engineer another processing layer to perform what’s called forced alignment on a second pass. This adds infrastructure and latency and also has more potential failure points. For example, they can potentially drift or fail entirely when trying to process rapid speech or speech buried by loud background noise.
Timestamped transcription, like Scribe can perform, sidesteps this issue by transcribing at a more granular level, word by word. It does this in a single API call. No additional infrastructure or scripting is required. The model calculates timestamps as it transcribes, so speech data always accurately mirrors audio.
How does timestamped, event-tagged transcription work?
Let’s use Scribe as an example.
A native word-level transcription model maps the speech sounds in an audio feed into a structured array of token objects. It labels every token as one of three types: word, spacing, or audio_event.
- word: A recognized, individual spoken word with noted start and end times and a speaker_id tag.
- spacing: Explicitly tagged silence or mid-speech pauses. Useful for helping captioning services to pace correctly. Not used for identified languages that do not use word spaces, such as Japanese, Mandarin, Thai, Lao, Burmese, and Cantonese.
- audio_event: Meaningful non-verbal audio, such as laughter or applause. The model knows to structure these as a separate event type and doesn’t force the audio into hallucinated speech.
Real-time streaming parameters
Word-level transcription models can also include special features to support live WebSocket audio transcription. For example:
- include_timestamps=true: This parameter makes the connection include word-level timestamps in the committed_transcript_with_timestamps JSON messages sent at the end of each finalized audio chunk, so you get timestamps while the stream is still running. Audio event tags are included only if tag_audio_events is also enabled.
- include_language_detection=true: This parameter tells the connection to tag different spoken languages detected along with the model’s confidence score for the detection. This helps you run multi-language captioning live.
How does a Speech to Text model tag non-speech events?
Scribe will tag audio events when you have the tag_audio_events parameter enabled. If so, it will tag and timestamp specific start and end times just like it does with words. It can tag a variety of reactions, with the most common being laughter and applause, but also other ambient sounds that might be contextually meaningful, including footsteps or background music.
At the most immediate level, this makes transcripts and captioning richer by adding important context. Sarcasm may be harder for a reader to intuit when reading simple, flat text. A [laughter] tag enriches the transcript and makes it more emotionally resonant.
Better analytics
Event tagging also makes any downstream analytics you run cleaner, as those tools don’t have to parse a single block of unstructured text. Native word-level transcription separates audio events as structured data so your tools can just filter them out when appropriate and run on purely verbal content.
Because spacing tokens make pauses visible, you can analyze them. This can be valuable for sentiment analysis and QA use cases, such as measuring how long a customer service agent was silent during a call.
Easier timestamp searching
Or you can specifically search them. If you’re running sentiment analysis on a product testing interview, you can specifically pull out meaningful emotional reactions. Or if you’re running a social account, you can quickly search for laughter in an hour-long speech to find the clips you think are most likely to go viral.
What are practical applications for structured timestamped transcripts?
Getting structured data natively in transcriptions has several important use cases.
Caption and subtitle generation
You can export timestamped transcripts directly into production captioning formats, including SRT and VTT, with no intermediary processing. Your video editing tool can use word timing to highlight words in the captions as they’re spoken.
The transcription model can easily parse rapid speech because it processes it word by word. Where a conventional model might stumble over rapid-fire speech or overlapping voices, a word-level model like Scribe can output a clean, structured transcription.
Audio and video search
Conventional transcription lets you match keywords in the text block, but without forced alignment, you can't jump to the moment a phrase was spoken. A keyword search on a word-level transcription is fully indexed and brings you right to the second a phrase was spoken in the audio. This effectively turns your audio archive into a fully searchable reference catalog. This capability is particularly useful for managing large media libraries, podcast networks, and corporate archives.
Compliance and risk auditing
Many times you’ll want to record audio containing sensitive information, for example, recording customer service calls for QA. These can be the calls most likely to contain sensitive or regulated personal information.
To stay compliant while still having audio data for analytics, you need a way to redact sensitive data from transcripts in real time. Native word-level timestamping lets you take advantage of features like ElevenLabs' Entity Detection, which flags sensitive entities (like credit card numbers or names) and returns their offsets and timestamps so you can obscure them in your transcripts as they stream in.
For example, you can detect and redact a credit card number, mother's maiden name, birthdate, and customer name as they're spoken on a call verifying their identity.
Automated social media clipping
You can programmatically automate keyword searches too to find highlights among longer-form content. For example, you could highlight important moments from a campaign speech or corporate seminar. This helps you turn recent content into professionally edited highlights for YouTube, LinkedIn, or TikTok.
Multichannel and multi-speaker timing
Scribe can transcribe up to five channels independently and in parallel. Each gets an assigned speaker ID and timestamping. This is more reliable than standard acoustic speaker diarization, which often struggles with back-and-forth dialogue. For multichannel recordings, Scribe’s speaker assignment is purely deterministic. That means Channel 0 maps to speaker_0, Channel 1 to speaker_1, and so on.
It can recognize speakers trying to talk simultaneously and can still generate independent timestamps for their speech. They can be in different languages too.
Exporting timestamped data in the format you need
If you want conventional transcriptions distributed in multiple formats, you likely will need to write the parsing scripts yourself. Scribe can export transcriptions in multiple formats depending on how you’re going to use them. You can find full schema definitions in theElevenLabs Create Transcript API Reference.
Structured data for developers: JSON
Scribe supports exporting to JSON for developers who want data in a schema they can pass to downstream applications. For example, developers can use this JSON data to build interactive media or a semantic search database for your organization. The full array of word, spacing, and audio_event tokens outputs with start and end offsets and speaker IDs.
Media captioning: SRT and VTT
Scribe can output SRT and VTT transcripts that you can feed directly into Adobe Premiere or other production applications without manual synchronization.
Human-readable: TXT, DOCX, and PDF
It can also output transcripts in reader-friendly formats. In these formats, Scribe strips out low-level timing markers to make the text more readable and suitable for legal or audit documentation. For example, this is useful when you need to efficiently distribute board meeting minutes immediately after a session, publish podcast transcripts for SEO, or archive legal records.
Start using ElevenLabs timestamped, event-tagged transcriptions
Native word-level transcription gives you the structured data you need for your workflows quickly and efficiently.
Get started today transcribing your audio with Scribe in ElevenAPI or learn more about word-level transcription.
