event-driven screen capture with platform-specific apis
Captures screen content from all connected monitors by listening to OS-level events (window focus changes, content updates) rather than polling continuously, using platform-specific graphics APIs: CoreGraphics on macOS, DXGI on Windows, and X11/PipeWire on Linux. This event-driven model reduces CPU usage by ~80% compared to continuous frame capture while maintaining temporal accuracy through configurable capture intervals (default 1 FPS). The VisionManager monitors trigger events and coordinates frame acquisition across multiple displays.
Unique: Uses event-driven capture triggered by OS-level window events rather than fixed-interval polling, reducing CPU by ~80% while maintaining temporal fidelity through platform-specific APIs (CoreGraphics, DXGI, X11/PipeWire) that integrate directly with OS event loops
vs alternatives: Achieves 80% lower CPU usage than continuous frame capture while maintaining multi-display support, unlike cloud-based screen recording services that require network bandwidth and introduce latency
multi-engine ocr text extraction from screen frames
Extracts text from every captured screen frame using platform-optimized OCR engines: Apple Vision framework on macOS, Windows native OCR on Windows, and Tesseract on Linux with fallback support. The system processes frames through a configurable OCR pipeline that handles multiple languages, variable text sizes, and rotated text. Extracted text is indexed alongside frame metadata (timestamp, bounding boxes, confidence scores) for later semantic search and retrieval.
Unique: Abstracts platform-specific OCR engines (Vision, Windows OCR, Tesseract) behind a unified interface with automatic fallback chains and confidence score normalization, enabling consistent text search across macOS, Windows, and Linux without user configuration
vs alternatives: Uses native OS OCR engines (Vision, Windows OCR) for faster processing than cloud-based alternatives like Google Cloud Vision, while maintaining local privacy and avoiding per-request API costs
multi-provider ai backend abstraction with local and cloud options
Abstracts AI service providers (OpenAI, Anthropic, Deepgram, local Whisper, local sentence-transformers) behind a unified configuration interface. Users can select which provider to use for each AI capability (transcription, embeddings, LLM reasoning) and switch between local and cloud options without code changes. The system includes fallback chains (e.g., try local Whisper first, fall back to Deepgram if unavailable) and usage tracking for cloud services. Configuration is stored in settings and can be updated via desktop app or API.
Unique: Provides a unified abstraction layer that allows users to configure and switch between local (Whisper, sentence-transformers) and cloud (OpenAI, Anthropic, Deepgram) AI providers per capability, with automatic fallback chains and usage tracking
vs alternatives: More flexible than single-provider solutions (Rewind.ai uses only cloud, local-only tools lack cloud option); enables cost optimization by mixing local and cloud processing based on use case
global keyboard shortcuts and system tray integration
Provides configurable global keyboard shortcuts (e.g., Cmd+Shift+P on macOS) to trigger Screenpipe actions from anywhere on the system, even when the desktop app is not focused. Shortcuts can open the search interface, pause/resume recording, or trigger custom Pipes. System tray integration provides quick access to Screenpipe status, recording state, and common actions. Shortcuts are registered at the OS level using platform-specific APIs (Cocoa on macOS, Win32 on Windows, X11 on Linux) and persist across app restarts.
Unique: Registers OS-level global keyboard shortcuts (Cocoa, Win32, X11) that work across all applications, enabling quick access to Screenpipe search and controls without switching windows; integrates system tray for status visibility
vs alternatives: Faster than opening desktop app or using REST API for quick actions; more discoverable than command-line shortcuts; system tray provides always-visible status unlike background-only services
privacy-preserving local-first architecture with optional encrypted cloud sync
Implements a privacy-first design where all data capture, processing, and storage occur locally on the user's device by default. Screen frames, audio, OCR results, and transcripts are stored in the local SQLite database and never transmitted to cloud services unless explicitly configured. Optional encrypted cloud sync can be enabled for backup and cross-device access, but encryption keys are managed locally and cloud provider cannot access unencrypted data. The system provides granular privacy controls (pause recording, exclude applications, redact sensitive data) and audit logs showing what data was captured and processed.
Unique: Implements local-first architecture where all data stays on device by default, with optional encrypted cloud sync where encryption keys are managed locally; provides granular privacy controls and audit logs for compliance
vs alternatives: More privacy-preserving than cloud-only services (Rewind.ai, Copilot for Windows) which transmit data to cloud; more flexible than local-only tools which lack backup options; compliant with GDPR and HIPAA by design
continuous audio transcription with voice activity detection
Transcribes system audio and microphone input using either local OpenAI Whisper or cloud-based Deepgram API, with integrated voice activity detection (VAD) to identify speech segments and reduce processing of silence. The audio pipeline captures raw PCM samples, applies VAD filtering to detect speech boundaries, batches audio chunks, and sends them to the transcription engine. Transcripts are timestamped and indexed alongside screen frames for synchronized search across audio and visual content.
Unique: Integrates voice activity detection to filter silence before transcription, reducing processing load by ~60% on typical office audio, and abstracts both local Whisper and cloud Deepgram backends with automatic fallback, enabling users to switch between privacy-first and speed-optimized modes
vs alternatives: Combines local VAD filtering with optional cloud transcription to reduce costs vs always-on cloud services, while maintaining privacy option via local Whisper; unlike Otter.ai or Rev, provides full control over transcription backend and audio data residency
semantic search across screen and audio history with vector embeddings
Enables full-text and semantic search across captured screen frames and audio transcripts by embedding text content into a vector database. The system extracts text from OCR results and transcripts, generates embeddings using configurable embedding models (local or cloud-based), and stores them in a local SQLite database with vector extension support. Search queries are embedded using the same model and matched against historical embeddings using cosine similarity, returning ranked results with temporal context (timestamps, associated frames, transcript segments).
Unique: Combines OCR text and audio transcripts into a unified vector embedding index stored locally in SQLite, enabling semantic search across both modalities without cloud transmission; supports pluggable embedding models (local sentence-transformers or cloud APIs) with automatic fallback
vs alternatives: Provides local semantic search without cloud dependency unlike Rewind.ai or Copilot for Windows, while supporting both screen and audio modalities in a single search index; faster than keyword-only search for paraphrased queries
rest api for programmatic access to captured data and search
Exposes a REST API that allows external applications and scripts to query captured screen frames, audio transcripts, and search results. The API provides endpoints for frame retrieval (by timestamp or ID), transcript search, semantic search, and metadata queries. The API is served by a local HTTP server (default port 3030) and supports authentication via API keys or local-only access. Responses include structured JSON with frame data (base64-encoded images, OCR text, timestamps), transcript segments, and search rankings.
Unique: Provides a local HTTP API (port 3030) that exposes both raw captured data (frames, transcripts) and AI-powered search (semantic search, OCR text) in a unified interface, enabling external tools to query personal activity history without cloud transmission
vs alternatives: Unlike cloud-based screen recording APIs (Rewind, Copilot for Windows), Screenpipe's REST API runs locally and provides direct access to raw data, enabling custom AI integrations without vendor lock-in; simpler than building custom database queries
+5 more capabilities