MediaPipe

Detects and localizes human faces in images and video streams using a lightweight neural network optimized for on-device inference, returning bounding boxes and confidence scores without requiring cloud connectivity. Implements hardware acceleration (GPU/NPU) on Android, iOS, and Web via platform-native APIs, enabling real-time processing at 30+ FPS on mobile devices with sub-100ms latency per frame.

Detects and tracks 21 hand keypoints (knuckles, joints, fingertips, palm center) in real-time video or images, enabling gesture recognition and hand pose estimation. Processes hand regions through a multi-stage pipeline: hand detection → hand cropping → landmark localization, with built-in support for left/right hand classification and multi-hand tracking across frames.

Generates images from text descriptions using a neural network-based generative model. Processes text prompts through a text encoder and diffusion model to produce novel images matching the description, supporting customization via negative prompts and generation parameters.

Enables fine-tuning of pre-trained MediaPipe models on custom datasets to adapt them for domain-specific tasks. Model Maker abstracts the training process, accepting labeled datasets and producing optimized models for deployment on Android, iOS, Web, or Python without requiring deep ML expertise.

Deploys trained or pre-trained MediaPipe models to Android, iOS, Web, and Python with automatic hardware acceleration (GPU, NPU) on supported devices. Abstracts platform-specific optimization details, providing a unified API surface across platforms while leveraging native hardware acceleration for real-time inference.

Provides a web-based interface (MediaPipe Studio) for visualizing, evaluating, and comparing MediaPipe models on images and videos without requiring code. Enables interactive testing of models, side-by-side comparison of different models or parameter configurations, and visualization of model outputs (bounding boxes, keypoints, masks, etc.).

Executes large language models (LLMs) on-device without cloud connectivity, enabling privacy-preserving text generation, completion, and reasoning tasks. Supports quantized or distilled LLM models optimized for mobile and edge devices, with configurable generation parameters (temperature, top-k, top-p, max tokens).

Enables running large language models (LLMs) on-device using MediaPipe's LLM Inference API. Supports quantized/compressed LLM models optimized for mobile and edge devices. Handles tokenization, inference, and token generation. Supports streaming token output for real-time text generation. Enables chatbots, text generation, and other LLM-based features without cloud calls. ARCHITECTURAL DETAILS UNKNOWN: documentation does not specify supported model formats, quantization methods, or provider support.

Generates images from text descriptions using a pre-trained text-to-image model. Takes text prompt as input and outputs generated image. ARCHITECTURAL DETAILS UNKNOWN: documentation does not specify model architecture, inference approach, or customization options. Likely uses a diffusion model or similar generative architecture optimized for mobile.

Detects and tracks 33 body keypoints (joints, landmarks across head, torso, arms, and legs) in images and video streams using a neural network-based approach. Outputs 3D coordinates (x, y, z) for each landmark with per-landmark visibility confidence, enabling pose estimation, fitness tracking, and motion analysis without cloud dependency.

Detects and localizes objects in images and video streams by identifying object categories and their spatial locations via bounding boxes. Supports multiple object detection models (COCO, Open Images, custom datasets) with configurable confidence thresholds, returning class labels, confidence scores, and bounding box coordinates for each detected object.

Segments images into semantic regions (pixel-level classification by category) or instance segments (individual object masks). Processes images through a neural network to produce dense pixel-level predictions, returning either per-pixel class labels (semantic) or per-object masks with instance IDs (instance segmentation).

Classifies text into predefined or custom categories using a neural network-based text encoder. Processes text through embedding and classification layers, returning predicted category labels with confidence scores. Supports fine-tuning on custom datasets via Model Maker for domain-specific classification tasks.

Converts text into fixed-size numerical embeddings (vectors) that capture semantic meaning, enabling similarity comparisons and semantic search. Uses a pre-trained text encoder model to transform variable-length text into a dense vector representation (e.g., 512-dimensional), where similar texts produce similar embeddings.

Identifies the language of input text by classifying it into one of 100+ supported languages. Uses a lightweight neural network classifier optimized for on-device inference, returning the detected language code (e.g., 'en', 'es', 'zh') with confidence score.

Classifies audio clips into predefined sound event categories (e.g., speech, music, applause, dog bark) using a neural network-based audio classifier. Processes audio spectrograms through a classification model, returning predicted event labels with confidence scores.

Enables users to refine image segmentation masks by providing interactive input (e.g., clicking to select regions, drawing strokes). Combines automated segmentation with user guidance to produce precise masks, using a neural network that accepts both image and user interaction as input.