Promptable Image Segmentation With Point And Box Inputs

via “interactive segmentation with user-guided mask refinement”

Google's cross-platform on-device ML framework with pre-built solutions.

Unique: Combines automated segmentation with interactive user refinement in a single API, enabling precise mask generation with minimal user effort; runs entirely on-device without cloud processing, making it suitable for privacy-sensitive image editing applications.

vs others: More user-friendly than fully automated segmentation for precise results, faster than manual pixel-by-pixel editing, but requires more user effort than fully automated alternatives and less feature-rich than professional image editing software like Photoshop.

via “bounding-box-prompt image segmentation with adaptive mask refinement”

Meta's foundation model for visual segmentation.

Unique: Encodes bounding boxes as dual corner points plus a learnable box token, allowing the same prompt encoder to handle points and boxes without separate branches. This design reuses the cross-attention mechanism, reducing model complexity while maintaining flexibility across prompt modalities.

vs others: More accurate than naive bounding box masking (e.g., connected components within box) because the transformer decoder understands object boundaries learned from 1.1B training images, handling occlusion and complex shapes within the box region.

via “bounding-box-based segmentation with automatic refinement”

Python AI package: segment-anything

Unique: Treats bounding boxes as prompts to the mask decoder rather than requiring box-specific training, enabling zero-shot box-to-mask conversion — unlike Mask R-CNN which requires end-to-end training with box and mask annotations

vs others: More flexible than Mask R-CNN for handling detection outputs from different models; enables refinement of detection boxes without retraining

* ⭐ 04/2023: [DINOv2: Learning Robust Visual Features without Supervision (DINOv2)](https://arxiv.org/abs/2304.07193)

Unique: Uses a two-stage architecture (image encoder + lightweight prompt decoder) that decouples image encoding from prompting, enabling amortized computation across multiple prompts on the same image. Unlike prior work (Mask R-CNN, DeepLab) that requires task-specific training, SAM's prompt-based design generalizes to arbitrary object categories through a unified decoder trained on 1.1B segmentation masks from diverse sources.

vs others: Faster and more flexible than interactive segmentation tools like Grabcut or GrabCut++ because it encodes the image once and reuses that encoding for multiple prompts, while maintaining zero-shot generalization across object categories without fine-tuning.