MineContext

Captures full-screen screenshots at configurable 5-second intervals via Electron's native screen capture APIs, storing raw image files to disk and queuing them for asynchronous VLM processing. The system uses a dedicated screenshot monitor thread that respects display state (active/idle) and integrates with the context capture pipeline to timestamp and batch screenshots for efficient processing without blocking the UI.

Processes captured screenshots through configurable VLM services (local or remote) to extract semantic descriptions of visual content, including detected activities, UI elements, text content, and contextual information. The system maintains a pluggable VLM client architecture supporting multiple providers (Doubao, OpenAI Vision, local models via Ollama) with fallback chains and caching of VLM responses to avoid redundant inference on duplicate frames.

Provides a cross-platform desktop UI built with Electron and React, managing application state through a centralized store (Redux or similar) with async middleware for backend API calls. The UI includes dashboard components for viewing summaries/todos/tips, search interface for context retrieval, settings panel for configuration, and real-time notifications for proactive content delivery. Electron main process handles window management, system tray integration, and native OS interactions.

Provides a REST API backend built with FastAPI and Python, exposing endpoints for context operations (capture, search, retrieval), consumption management (summaries, todos, tips), and configuration. The backend uses async/await for non-blocking I/O, integrates with background task queues (Celery, RQ) for long-running operations, and maintains SQLite and vector database connections. API is served on localhost:1733 by default with CORS enabled for Electron frontend.

Defines a unified context schema supporting multiple context types (screenshots, documents, activities, todos, tips, summaries) with common metadata (timestamp, source, type, embeddings) and type-specific fields. The system maintains context type definitions in code and database schema, enabling polymorphic queries that treat different context types uniformly while preserving type-specific information. Context merging logic combines related items (e.g., multiple screenshots of same activity) into higher-level abstractions.

Tracks user activity by analyzing captured context (screenshots, documents, interactions) and extracting activity records with temporal boundaries (start time, end time, duration). The system maintains a temporal index enabling efficient queries by time range, activity type, and duration. Activity records include metadata (application/document name, activity description, confidence score) and references to source context items.

Stores captured context in a dual-database architecture: SQLite for structured metadata (timestamps, activity types, document references) and ChromaDB/Qdrant for vector embeddings enabling semantic similarity search. The system maintains a unified schema across both stores with automatic synchronization, allowing queries to combine structured filters (date range, activity type) with semantic search (find similar activities) in a single operation.

Converts text descriptions from VLM analysis and document content into high-dimensional embeddings (768-1536 dimensions) using configurable embedding models (local or remote). The system maintains an embedding client with provider abstraction, supporting multiple backends (Doubao embeddings, OpenAI embeddings, local models via Ollama) with batch processing for efficiency and caching to avoid recomputing embeddings for identical text.

Automatically generates daily activity summaries by querying the context database for all activities within a 24-hour window, processing them through an LLM with a structured prompt template, and storing the summary as a consumable artifact. The system uses a scheduler (APScheduler or similar) to trigger summary generation at configurable times (default 08:00), with fallback to manual regeneration and debug mode for prompt refinement.

Analyzes captured context (screenshots, documents, activity descriptions) to automatically extract actionable todos and tasks using LLM-based extraction with structured prompts. The system runs extraction at configurable intervals (default 1800s/30 minutes), deduplicates extracted todos against existing items, and stores them with metadata (source context, extraction confidence, priority) for consumption in the UI.

Generates contextually relevant tips and suggestions by analyzing recent activity patterns and context, using LLM-based generation with prompt templates that reference detected activities and user behavior. The system runs tip generation at configurable intervals (default 3600s/1 hour), filters tips for relevance using embedding similarity, and stores them with metadata for proactive delivery to the user.

Enables retrieval of relevant past context using hybrid search combining vector similarity (semantic search) with structured filters (time range, activity type, source). The system queries the vector database for semantically similar context items and applies SQLite filters to narrow results, returning ranked results with relevance scores. This supports both programmatic API access and UI-based search interfaces.

Accepts user-uploaded documents (PDF, DOCX, images, etc.) and processes them through a unified pipeline: file type detection, content extraction (text via OCR or parsing), VLM analysis for visual content, embedding generation, and storage in dual databases. The system maintains a document vault with metadata (upload time, file type, source) and integrates documents into the context search and retrieval system.

Provides a pluggable LLM client architecture supporting multiple providers (OpenAI, Anthropic, local Ollama, custom endpoints) with automatic fallback chains and provider-specific configuration. The system maintains a provider registry, handles API authentication, manages rate limiting, and implements retry logic with exponential backoff. Configuration is stored in YAML/JSON files with UI-based settings management.