AssemblyAI

Converts pre-recorded audio files to text using Universal-3 Pro or Universal-2 deep learning models trained on 12.5+ million hours of audio. Processes audio asynchronously via REST API, returning word-level timestamps, automatic punctuation/casing, and language detection across 99 languages (Universal-2) or 6 primary languages (Universal-3 Pro). Supports custom spelling dictionaries and keyterm prompting (up to 1000 phrases, 6 words max per phrase) to improve domain-specific accuracy.

Transcribes live audio streams in real-time using Universal-3 Pro Streaming model with ultra-low latency (specific latency metrics not documented). Provides interim transcription management (ITM) for progressive text updates, automatic punctuation/casing, end-of-turn detection, and speaker identification by name or role. Integrates with LiveKit SDK and Pipecat framework for voice agent applications. Processes audio chunks via WebSocket or streaming REST API with continuous output.

Returns precise word-level timing information for each word in the transcript, enabling synchronization with video, highlighting, or interactive playback. Operates as a built-in feature of both pre-recorded and streaming transcription APIs, returning start and end timestamps (in milliseconds or seconds) for each word. Enables precise word-level seeking in audio/video players and transcript-to-media synchronization.

Detects and labels non-speech audio events (background noise, music, silence, beeps, etc.) within transcripts, annotating them with tags like '[MUSIC]', '[BEEP]', '[SILENCE]' or similar markers. Operates as a built-in feature of transcription APIs that identifies acoustic events and inserts event markers into the transcript at appropriate positions. Enables accurate transcription of audio with mixed content (speech + music + sound effects).

Detects and captures disfluencies, filler words, and informal speech patterns in transcripts, including: fillers (um, uh, er, erm, ah, hmm, mhm, like, you know, I mean), repetitions, restarts, stutters, and informal speech markers. Operates as a built-in feature of transcription APIs that identifies these patterns and optionally includes them in the transcript or flags them separately. Enables analysis of speech fluency, speaker confidence, and communication patterns.

Provides native Python and JavaScript SDKs for easy integration with AssemblyAI transcription APIs, supporting async/await patterns for non-blocking API calls. SDKs abstract REST API complexity, handle authentication, manage polling for async transcription jobs, and provide type-safe interfaces. Enables developers to integrate transcription into applications without manual HTTP request handling or webhook management.

Provides pre-built integrations with LiveKit (WebRTC media server) and Pipecat (voice agent framework) for building real-time voice agents and conversational AI applications. Integrations handle streaming audio transport, transcription, and response generation without custom WebSocket or streaming protocol implementation. Enables rapid voice agent development by combining AssemblyAI transcription with LiveKit media handling and Pipecat orchestration.

Provides Model Context Protocol (MCP) integration enabling AI coding agents (e.g., Claude) to call AssemblyAI transcription capabilities as tools. Allows AI agents to transcribe audio, extract entities, and analyze speech content as part of multi-step reasoning and planning workflows. Integrates with Claude and other MCP-compatible AI models for agentic transcription use cases.

Automatically detects and extracts structured entities (person names, company names, email addresses, dates, locations) from transcribed audio during speech-to-text processing. Operates as a built-in feature of both pre-recorded and streaming transcription APIs without separate API calls. Returns entity spans with word-level positions in the transcript, enabling precise entity linking and downstream data extraction workflows.

Segments transcripts by speaker and labels each utterance with speaker identity. For pre-recorded audio, uses speaker diarization to automatically identify speaker boundaries and assign speaker labels (Speaker 1, Speaker 2, etc.). For streaming, supports explicit speaker identification by name or role provided via API parameters. Operates at the utterance level, returning speaker labels alongside transcript segments with word-level timing.

Detects and redacts personally identifiable information (PII) and potentially harmful content from transcripts. Operates as a post-transcription feature that identifies sensitive data patterns (credit card numbers, social security numbers, phone numbers, email addresses, etc.) and either redacts them or flags them for review. Integrates with compliance and privacy workflows to ensure transcripts meet regulatory requirements (HIPAA, GDPR, PCI-DSS).

Generates abstractive summaries of transcribed audio content, extracting key insights, action items, and main topics. Operates as a post-transcription feature that analyzes the full transcript and produces concise summaries suitable for meeting notes, call summaries, or content indexing. Technical implementation details (model architecture, summary length, customization options) not documented.

Analyzes emotional tone and sentiment polarity of transcribed speech, detecting positive, negative, or neutral sentiment at the utterance or conversation level. Operates as a post-transcription feature that processes transcript text and returns sentiment labels with confidence scores. Technical implementation (model architecture, granularity level, emotion categories) not documented.

Improves transcription accuracy for domain-specific terminology by providing up to 1000 custom phrases (max 6 words per phrase) that the model should prioritize during decoding. Operates as a parameter in the transcription API request, biasing the speech recognition model toward expected vocabulary without requiring model fine-tuning. Included with Universal-3 Pro transcription; not supported on Universal-2.

Allows developers to provide natural language instructions to control transcription behavior, inject context, or specify expected content patterns without modifying model weights. Operates as a beta feature on Universal-3 Pro that accepts free-form text prompts describing transcription preferences (e.g., 'This is a medical conversation, prioritize medical terminology' or 'Speaker is discussing financial markets'). Costs $0.05/hour additional on top of base transcription pricing.

Specialized transcription mode optimized for healthcare conversations, medical terminology, and clinical documentation. Operates as an add-on feature ($0.15/hour) available on both Universal-3 Pro and Universal-2 models, improving accuracy for medical terms, drug names, anatomical references, and healthcare-specific language patterns. Implementation details (model architecture, training data, terminology coverage) not documented.