AssemblyAI

Converts pre-recorded audio files to text using Universal-3 Pro or Universal-2 models via asynchronous REST API processing. Universal-3 Pro achieves market-leading accuracy across 6 languages (English, Spanish, German, French, Italian, Portuguese) with context-aware prompting; Universal-2 supports 99 languages at lower cost. Processing returns word-level timestamps, speaker segmentation, and confidence scores via polling or webhook callbacks.

Processes live audio streams via WebSocket or streaming protocol, delivering near-real-time transcription with word-level timestamps and speaker diarization. Uses Universal-3 Pro Streaming model with same context-aware prompting and entity detection as pre-recorded variant. Designed for live call transcription, voice conference capture, and real-time voice agent interactions.

Automatically generates summaries of transcribed conversations and extracts key insights including action items, decisions, topics discussed, and sentiment trends. Summarization works on full transcripts or conversation segments. Returns structured summaries with configurable detail levels (brief, detailed, executive summary). Claimed in artifact description but detailed implementation unknown.

Analyzes emotional tone and sentiment in transcribed conversations, detecting speaker sentiment (positive, negative, neutral) and emotional states (anger, frustration, satisfaction, etc.). Returns sentiment scores per speaker, conversation segment, or overall. Enables customer satisfaction measurement, agent performance evaluation, and conversation quality assessment.

Provides precise word-level timestamps for every word in the transcript, enabling exact audio segment retrieval and temporal alignment with video or other media. Timestamps are returned in milliseconds with confidence scores. Enables video subtitle generation, audio clip extraction, and precise quote verification.

Specialized transcription mode optimized for medical conversations including clinical terminology, drug names, medical procedures, and patient information. Uses domain-specific language model tuning and medical vocabulary injection. Adds $0.15/hour to transcription cost. Supports both Universal-3 Pro and Universal-2 models.

Official SDKs for Python and JavaScript enable developers to integrate AssemblyAI transcription into applications without building raw HTTP clients. SDKs provide type-safe API bindings, automatic retry logic, error handling, and streaming support. Integrations with LiveKit and Pipecat frameworks enable voice agent and real-time communication use cases.

Provides Model Context Protocol (MCP) integration enabling AI agents and LLMs to access AssemblyAI transcription capabilities through a standardized interface. Documentation available at `/llms.txt` and `/llms-full.txt` endpoints. Enables agents to transcribe audio, extract insights, and perform speech understanding tasks as part of multi-step reasoning workflows.

Automatically detects and segments audio by speaker, labeling distinct speakers (Speaker A, Speaker B, etc.) with timestamps for when each speaker begins and ends. Works across both pre-recorded and streaming APIs. Adds $0.02/hour to transcription cost. Enables speaker role assignment via prompting (e.g., 'Speaker 1 is the customer, Speaker 2 is the agent').

Automatically extracts and labels named entities from transcribed speech including person names, company names, email addresses, phone numbers, dates, and locations. Works on both pre-recorded and streaming transcripts. Returns entity type, text, and timestamp for each detected entity. Enables domain-specific entity detection via custom keyterms prompting.

Identifies and labels filler words (um, uh, like, you know) and speech disfluencies (stutters, repetitions, restarts, informal speech patterns) in transcripts. Marks these elements in the transcript output with special tags (e.g., `[um]`, `[uh]`) and provides word-level classification. Useful for speech quality analysis, speaker coaching, and conversation naturalness scoring.

Detects and tags non-speech audio events in transcripts such as background noise, music, silence, and other acoustic events. Marks these events with special tags (e.g., `[beep]`, `[music]`, `[silence]`) at the appropriate timestamps in the transcript. Useful for audio quality assessment, content moderation, and transcript cleanup.

Enables domain-specific vocabulary injection and context guidance via natural language prompts and keyterms lists. Developers provide up to 1000 custom words/phrases (max 6 words per phrase) and optional context prompts (e.g., 'This is a medical consultation') to improve transcription accuracy for specialized terminology. Works with Universal-3 Pro ($0.05/hr add-on) and pre-recorded transcription.

Provides a proprietary end-to-end voice agent stack built on streaming speech-to-text, enabling developers to build conversational voice agents without managing separate STT, NLU, and TTS components. Agents handle real-time audio input/output, speaker identification, and conversation state management. Priced at $4.50/hour of audio. Described as 'fastest path to a working voice agent' with production-ready reliability.

Automatically detects and redacts personally identifiable information (PII) from transcripts including names, email addresses, phone numbers, social security numbers, credit card numbers, and other sensitive data. Redaction can be applied to transcript text (replacing with `[PII]` or similar) or audio (via beep/silence masking). Enables compliance with data privacy regulations (GDPR, HIPAA, CCPA).

Automatically detects and flags content policy violations in transcripts including profanity, hate speech, harassment, and other prohibited content. Returns moderation scores and violation categories for each detected segment. Enables content filtering for compliance, brand safety, and user experience management.