joinly

Enables AI agents to join Google Meet, Zoom, and Microsoft Teams meetings through Playwright-based browser automation with platform-specific controllers that handle each platform's unique UI patterns, authentication flows, and meeting state management. The BrowserMeetingProvider abstracts platform differences while delegating to GoogleMeetController, ZoomController, and TeamsController for platform-specific interactions, managing virtual display (Xvfb) and audio device routing.

Captures audio from meeting participants in real-time through PulseAudio integration and applies Voice Activity Detection (VAD) to filter silence and background noise before sending to transcription. The DefaultTranscriptionController orchestrates the VAD → STT pipeline, using pluggable VAD service providers (local or cloud-based) to reduce transcription costs by only processing segments with actual speech.

Provides high-level Python SDK (joinly-client package) with JoinlyClient class that abstracts MCP communication and session management, enabling developers to build meeting agents without understanding MCP protocol details. SDK handles connection lifecycle, tool calling, and transcript streaming, providing a simple async API for agent code.

Defines shared data types (Transcript, AudioFormat, AudioChunk) and service provider protocols in joinly-common package, ensuring consistent interfaces across server and client packages. Protocols define expected behavior for VAD, STT, and TTS providers, enabling type-safe provider implementations and reducing integration errors.

Converts filtered audio segments to text using configurable STT service providers (e.g., OpenAI Whisper, Google Cloud Speech, local models). The DefaultTranscriptionController receives VAD-filtered audio chunks and routes them to the selected STT provider, returning Transcript objects with text, confidence scores, and timing metadata for agent consumption.

Converts agent text responses to speech and outputs audio to the meeting in real-time using configurable TTS service providers (e.g., Resemble, Google Cloud TTS, local TTS engines). The DefaultSpeechController manages the TTS → audio output pipeline, handling audio format conversion, buffering, and PulseAudio device routing to ensure agent speech is heard by meeting participants.

Exposes meeting capabilities (join, transcribe, speak, get participants, etc.) as standardized Model Context Protocol (MCP) tools that LLM agents can call. The FastMCP server interface wraps meeting operations as callable tools with JSON schemas, enabling any MCP-compatible LLM client to interact with meetings through a standard protocol without needing to understand Joinly's internal APIs.

Manages meeting session lifecycle (creation, state tracking, resource cleanup) through the MeetingSession orchestrator class, using dependency injection to wire together platform providers, audio controllers, and service implementations. Sessions maintain state across multiple operations, handle concurrent audio processing, and ensure proper resource cleanup on meeting termination.

Provides ConversationalToolAgent class that wraps LLM integration for building meeting agents that can understand meeting context, call MCP tools, and generate responses. The agent maintains conversation history, handles tool calling loops, and integrates with any LLM provider that supports function calling (OpenAI, Anthropic, local models via Ollama).

Provides pluggable service provider architecture for VAD, STT, and TTS, allowing runtime selection and configuration without code changes. Service providers are registered in a dependency injection container, enabling easy swapping between local and cloud implementations based on deployment environment (privacy requirements, cost, latency).

Streams transcripts from meetings to connected clients in real-time using Server-Sent Events (SSE), including timing information (start_time, end_time) and speaker metadata. The Transcript data type (from joinly-common) standardizes transcript format across all STT providers, enabling consistent agent consumption regardless of backend.

Provides Docker containerization with pre-configured Xvfb virtual display, PulseAudio daemon, and Playwright browser for headless meeting automation. Multiple Docker image variants support different deployment scenarios (minimal, full-featured, GPU-accelerated), with environment variable configuration for service providers and meeting parameters.