Segment Anything 2

Accepts single or multiple point coordinates on an image and generates precise object segmentation masks using a vision transformer encoder paired with a lightweight mask decoder. The architecture encodes the image once, then efficiently processes point prompts through a prompt encoder that converts coordinates to embeddings, which are fused with image features via cross-attention mechanisms to produce per-pixel segmentation logits.

Accepts bounding box coordinates (top-left and bottom-right corners) and generates segmentation masks by encoding the box as corner point embeddings plus a special box token, then fusing these with image features through cross-attention. The decoder refines the mask iteratively to respect box boundaries while capturing fine object details within the box region.

Provides a unified interface for loading pre-trained SAM2 checkpoints in multiple sizes (Tiny 38.9M, Small 46M, Base-Plus 80.8M, Large 224.4M parameters) from local files or Hugging Face Hub, with automatic architecture instantiation and weight loading. The system handles checkpoint versioning, device placement (CPU/GPU), and optional quantization for memory efficiency.

Supports batch processing of multiple images or video frames through a single forward pass, with dynamic batching that groups inputs of similar sizes to maximize GPU utilization. The system uses memory pooling to reuse allocated tensors across batch items, reducing allocation overhead and enabling efficient processing of large image collections.

Estimates prediction confidence for each segmentation mask through multiple mechanisms: predicted IoU (intersection-over-union with ground truth, estimated by the model), stability score (mask consistency under input perturbations), and logit magnitude. These scores enable filtering unreliable predictions and ranking masks by confidence, supporting downstream applications that require quality thresholds.

Accepts a previous segmentation mask (binary or soft) as input and refines it by encoding the mask as a spatial feature map, concatenating it with image features, and passing through the decoder to produce an improved mask. Supports iterative refinement where outputs from one iteration become inputs to the next, enabling progressive segmentation correction through multiple rounds.

Generates comprehensive segmentation masks for all objects in an image without user prompts by systematically sampling point grids across the image, running inference for each point, and merging overlapping masks using IoU-based deduplication. The SAM2AutomaticMaskGenerator class orchestrates this process, filtering low-confidence masks and returning a set of non-overlapping masks covering the entire image.

Tracks multiple objects through video sequences by maintaining a streaming memory buffer of encoded features from previous frames, using cross-frame attention to propagate object masks forward in time. The SAM2VideoPredictor processes frames sequentially, storing compressed representations of segmented objects in memory, then uses these memories to predict masks in subsequent frames without re-encoding the entire history, enabling real-time processing.

Extends video tracking to handle multiple objects simultaneously by maintaining separate memory streams for each tracked object, allowing independent prompts (points, boxes, masks) per object in the first frame. The system tracks each object through subsequent frames using dedicated memory buffers, enabling multi-object segmentation without object ID conflicts or cross-object interference.

Provides SAM2VideoPredictorVOS, a specialized video predictor that wraps the base model with torch.compile() for graph-level optimization, reducing memory overhead and increasing throughput for video object segmentation (VOS) tasks. The optimization targets the streaming memory update and mask decoding loops, which are the computational bottlenecks in frame-by-frame processing.

Encodes input images using a Vision Transformer (ViT) backbone that produces multi-scale hierarchical features through intermediate layer outputs, capturing both global semantic context and local spatial details. The encoder processes images at a fixed resolution (e.g., 1024×1024), producing feature pyramids that are used by both the mask decoder and memory systems for efficient cross-attention.

Decodes segmentation masks from image features and prompt embeddings using a lightweight transformer decoder with iterative refinement, where each iteration refines the mask prediction by re-attending to image features and previous mask predictions. The decoder uses a small number of transformer blocks (2-4) to keep inference latency low while maintaining accuracy through multiple refinement iterations.

Fuses image features with prompt embeddings (from points, boxes, or masks) using cross-attention mechanisms, where prompt embeddings attend to image features to identify relevant regions, and image features are updated based on prompt context. This fusion enables the decoder to focus on prompt-relevant image regions, improving segmentation accuracy and enabling multi-prompt composition.