xiaozhi-esp32-server

Implements a persistent WebSocket connection handler (ConnectionHandler class) that manages per-client session state, routes incoming audio frames at 60ms intervals via AudioRateController, and maintains bidirectional communication with ESP32 hardware. Uses frame-based timing synchronization to ensure consistent audio delivery rates and handles connection lifecycle events (hello handshake, authentication, disconnection). The architecture supports multiplexed concurrent device connections through async I/O patterns.

Integrates pluggable ASR providers (FunASR, Whisper, etc.) that process streaming audio frames in real-time, converting spoken input to text through provider-specific APIs. The system buffers incoming audio, detects speech boundaries via SileroVAD (Voice Activity Detection), and routes complete utterances to the configured ASR provider. Supports both cloud-based (OpenAI Whisper, Alibaba FunASR) and on-device (local Silero models) recognition with configurable fallback chains.

Implements centralized configuration loading from YAML files (config.yaml) that define AI providers (LLM, ASR, TTS), model parameters, device settings, and system behavior. The system supports environment variable substitution for sensitive data (API keys), configuration validation against schema, and hot-reload capabilities for non-critical settings. Configurations are hierarchically organized (global, per-user, per-device) with inheritance and override rules. Integrates with database for user-specific configuration overrides.

Implements real-time voice activity detection using Silero VAD model, which processes streaming audio frames to identify speech boundaries (start/end of utterance). The system runs VAD on incoming audio, buffers frames until speech ends, and triggers ASR only on complete utterances. Silero VAD is lightweight (~40MB) and runs on CPU, making it suitable for edge deployment. Supports configurable sensitivity and frame-based processing at 16kHz sample rate.

Provides a plugin architecture that allows developers to create custom functions in Python and register them with the function registry for invocation via intent recognition. Plugins are stored in plugins_func directory, automatically discovered and loaded at startup, and can access system context (user_id, device_id, conversation history). Each plugin is a Python function with type hints and docstring documentation, which are automatically converted to JSON Schema for parameter validation. Supports both synchronous and asynchronous function execution with error handling and result serialization.

Provides pluggable TTS providers (Azure, Google Cloud, ElevenLabs, local TTS engines) that convert text responses into audio streams, with support for voice cloning and custom voice parameters. The system accepts text input from LLM responses, applies provider-specific voice selection and prosody controls, streams audio back to ESP32 clients in 60ms frames, and manages voice profile storage for user-specific voice preferences. Supports both streaming TTS (real-time audio generation) and batch synthesis with caching.

Processes LLM-generated intent outputs through a function registry that maps recognized intents to executable Python functions or MCP tool calls. The system parses LLM responses for intent names and parameters, validates them against a schema registry, and executes corresponding plugins (built-in or user-defined) with automatic error handling and result serialization. Supports both synchronous function calls and async task queuing for long-running operations. Integrates with MCP (Model Context Protocol) for standardized tool definitions.

Maintains per-user conversation history with configurable context windows, storing previous user utterances, assistant responses, and execution results in a structured format. The system passes relevant context to the LLM for each turn, implements sliding-window context truncation to manage token budgets, and supports memory persistence across sessions via database storage. Integrates with knowledge base (RAG) to augment context with relevant documents and maintains dialogue state (current topic, user preferences, device state).

Provides a knowledge base management system that stores documents, generates embeddings, and performs semantic search to augment LLM context. The system accepts document uploads (PDF, TXT, Markdown), chunks them into semantic segments, generates embeddings via configured embedding models, and stores them in a vector database. During conversation, relevant documents are retrieved based on semantic similarity to user queries and injected into the LLM prompt. Supports multiple embedding providers (OpenAI, local models) and vector databases (Milvus, Weaviate, Pinecone).

Abstracts multiple LLM providers (OpenAI, Anthropic, Alibaba, local models) through a unified interface, allowing configuration-based provider selection and automatic fallback to secondary providers on failure. The system manages API keys, model parameters (temperature, max_tokens), and prompt formatting for each provider, implements retry logic with exponential backoff, and tracks provider health/availability. Supports both streaming (for real-time response generation) and batch LLM calls with configurable timeout handling.

Provides a management console for registering ESP32 devices, tracking firmware versions, and delivering OTA updates. The system maintains a device registry with device metadata (device_id, firmware_version, last_seen, user_binding), stores firmware binaries in cloud storage, and implements a secure update protocol that validates checksums and version compatibility before deployment. Supports staged rollouts (percentage-based deployment) and rollback to previous versions if updates fail. Integrates with the web management console for user-facing device management.

Provides a web UI (manager-web) and REST API (manager-api) for user management, device binding, model configuration, and knowledge base administration. The system implements JWT-based authentication, role-based access control (RBAC), and per-user device isolation. Users can bind ESP32 devices to their accounts, configure LLM/ASR/TTS providers, manage voice profiles, upload knowledge base documents, and monitor device status. Built with Spring Boot backend and Vue.js frontend, backed by MySQL database and Redis cache.

Provides MQTT protocol support for integrating with smart home platforms (Home Assistant, OpenHAB, Zigbee2MQTT) by publishing device state changes and subscribing to control commands. The system maintains MQTT topic hierarchies for device state (e.g., /xiaozhi/device/{device_id}/state), translates between Xiaozhi protocol messages and MQTT payloads, and implements bidirectional synchronization. Supports both publish-subscribe patterns for state updates and request-response patterns for command execution.