Screenpipe

Captures screen content from all connected monitors by listening to OS-level events (window focus changes, content updates) rather than polling continuously, using platform-specific graphics APIs: CoreGraphics on macOS, DXGI on Windows, and X11/PipeWire on Linux. This event-driven model reduces CPU usage by ~80% compared to continuous frame capture while maintaining temporal accuracy through configurable capture intervals (default 1 FPS). The VisionManager monitors trigger events and coordinates frame acquisition across multiple displays.

Extracts text from every captured screen frame using platform-optimized OCR engines: Apple Vision framework on macOS, Windows native OCR on Windows, and Tesseract on Linux with fallback support. The system processes frames through a configurable OCR pipeline that handles multiple languages, variable text sizes, and rotated text. Extracted text is indexed alongside frame metadata (timestamp, bounding boxes, confidence scores) for later semantic search and retrieval.

Abstracts AI service providers (OpenAI, Anthropic, Deepgram, local Whisper, local sentence-transformers) behind a unified configuration interface. Users can select which provider to use for each AI capability (transcription, embeddings, LLM reasoning) and switch between local and cloud options without code changes. The system includes fallback chains (e.g., try local Whisper first, fall back to Deepgram if unavailable) and usage tracking for cloud services. Configuration is stored in settings and can be updated via desktop app or API.

Provides configurable global keyboard shortcuts (e.g., Cmd+Shift+P on macOS) to trigger Screenpipe actions from anywhere on the system, even when the desktop app is not focused. Shortcuts can open the search interface, pause/resume recording, or trigger custom Pipes. System tray integration provides quick access to Screenpipe status, recording state, and common actions. Shortcuts are registered at the OS level using platform-specific APIs (Cocoa on macOS, Win32 on Windows, X11 on Linux) and persist across app restarts.

Implements a privacy-first design where all data capture, processing, and storage occur locally on the user's device by default. Screen frames, audio, OCR results, and transcripts are stored in the local SQLite database and never transmitted to cloud services unless explicitly configured. Optional encrypted cloud sync can be enabled for backup and cross-device access, but encryption keys are managed locally and cloud provider cannot access unencrypted data. The system provides granular privacy controls (pause recording, exclude applications, redact sensitive data) and audit logs showing what data was captured and processed.

Transcribes system audio and microphone input using either local OpenAI Whisper or cloud-based Deepgram API, with integrated voice activity detection (VAD) to identify speech segments and reduce processing of silence. The audio pipeline captures raw PCM samples, applies VAD filtering to detect speech boundaries, batches audio chunks, and sends them to the transcription engine. Transcripts are timestamped and indexed alongside screen frames for synchronized search across audio and visual content.

Enables full-text and semantic search across captured screen frames and audio transcripts by embedding text content into a vector database. The system extracts text from OCR results and transcripts, generates embeddings using configurable embedding models (local or cloud-based), and stores them in a local SQLite database with vector extension support. Search queries are embedded using the same model and matched against historical embeddings using cosine similarity, returning ranked results with temporal context (timestamps, associated frames, transcript segments).

Exposes a REST API that allows external applications and scripts to query captured screen frames, audio transcripts, and search results. The API provides endpoints for frame retrieval (by timestamp or ID), transcript search, semantic search, and metadata queries. The API is served by a local HTTP server (default port 3030) and supports authentication via API keys or local-only access. Responses include structured JSON with frame data (base64-encoded images, OCR text, timestamps), transcript segments, and search rankings.

Provides a plugin architecture called 'Pipes' that allows users to write custom automations triggered by screen and audio events. Pipes are JavaScript/TypeScript functions that receive captured frames, transcripts, and search results as input and can execute actions (send notifications, trigger webhooks, modify system state). The system includes a component registry for reusable UI elements and integrates with the MCP (Model Context Protocol) server for LLM-powered automations. Pipes are executed in a sandboxed runtime with access to Screenpipe's data APIs.

Implements a Model Context Protocol server that exposes Screenpipe's data (frames, transcripts, search results) as tools and resources to LLMs and AI agents. The MCP server allows Claude, GPT, and other LLMs to query screen history, search for content, and retrieve context about past activities. The server translates LLM tool calls into Screenpipe API requests and returns structured results. This enables AI agents to use Screenpipe as a memory system for decision-making and reasoning.

Implements an autonomous AI agent called 'Pi' that can observe screen content, understand UI elements, and execute tasks by simulating user interactions (mouse clicks, keyboard input, form filling). The agent uses vision-language models to interpret screen state, reason about next steps, and generate actions. Pi integrates with Screenpipe's frame capture and OCR to understand current UI state, and can chain multiple actions to complete multi-step workflows (e.g., filling out forms, navigating websites, running terminal commands).

Stores all captured data (frames, OCR text, transcripts, metadata) in a local SQLite database with full-text search (FTS5) and vector search extensions. The database schema includes tables for frames (with base64 image data), OCR results (text and bounding boxes), transcripts (with word-level timestamps), and embeddings (vector representations for semantic search). Indexes are automatically maintained as new data arrives. The database is stored locally on disk (no cloud sync by default) and can be queried via SQL or through Screenpipe's REST API.

Provides a Tauri-based desktop application (macOS, Windows, Linux) with a visual timeline interface for browsing captured screen history. The timeline displays thumbnail previews of captured frames chronologically, allowing users to scrub through time and view associated OCR text and transcripts. The 'Rewind' feature enables quick playback of screen activity at accelerated speed. The UI includes search interface for querying captured data, settings panel for configuring capture and AI backends, and system tray integration for quick access. The application communicates with the local Screenpipe server via REST API.