GPT Researcher

Decomposes user research queries into structured sub-queries using a dedicated planner agent that analyzes the original task, identifies knowledge gaps, and generates parallel search queries. The system uses a three-tier LLM strategy (fast model for planning, standard for execution, advanced for synthesis) to balance cost and quality. Sub-queries are executed in parallel across multiple retrievers, with results aggregated and deduplicated before synthesis.

Executes parallel web scraping across multiple URLs identified by search retrievers, using a browser skill that handles dynamic content, JavaScript rendering, and anti-bot detection. The system validates source credibility, filters irrelevant content, and extracts structured information (text, metadata, citations). Results are cached and deduplicated to avoid redundant scraping. Supports domain filtering to prioritize authoritative sources and exclude low-quality domains.

Provides multiple frontend options: NextJS production frontend with full state management and history tracking, vanilla JavaScript lightweight frontend for minimal dependencies, and embed script for integration into third-party websites. Frontends manage research state (queries, results, reports), maintain execution history, and provide interactive controls (start/pause/cancel research). The embed script enables drop-in integration without backend modifications. All frontends communicate with the FastAPI backend via REST or WebSocket APIs.

Implements domain filtering to prioritize authoritative sources and exclude low-quality domains. The curator skill evaluates source credibility using configurable rules (domain reputation, content quality, citation count, etc.). Filtering can be applied at retrieval time (to reduce noise) or post-retrieval (to validate sources). The system maintains a configurable domain whitelist/blacklist and can be extended with custom credibility scoring functions. Results are ranked by credibility score, enabling users to prioritize high-quality sources.

Integrates image generation (DALL-E, Midjourney, Stable Diffusion, etc.) to create illustrations for research reports. The system generates image prompts based on report content, calls image generation APIs, and embeds results in final reports. Supports configurable image generation backends and can be disabled for cost optimization. Generated images are cached to avoid redundant generation. The system can generate images for key concepts, data visualizations, or report sections.

Implements a flexible configuration system supporting environment variables, YAML/JSON config files, and runtime parameter overrides. The Config class centralizes all configuration (LLM providers, retrievers, research modes, etc.) with sensible defaults. Configuration can be loaded from multiple sources with precedence (environment > config file > defaults). Supports configuration validation and schema enforcement. Enables per-deployment customization without code changes.

Persists research tasks and execution history to enable task resumption, state recovery, and audit trails. The system stores task metadata (query, configuration, results), execution logs, and intermediate states. Supports querying research history, retrieving previous reports, and resuming interrupted research. State is stored in configurable backends (database, file system, cloud storage). Enables users to track research evolution and compare results across different configurations.

Manages research context across multiple sources using a context manager skill that compresses information to fit within LLM token limits while preserving semantic meaning. The system tracks citations for each piece of information, maintains source provenance, and synthesizes findings into coherent narratives. Uses sliding-window context management to handle large research datasets, with configurable compression strategies (summarization, extraction, embedding-based filtering) to optimize token usage while maintaining factual accuracy.

Generates research reports in three configurable modes (standard, detailed, deep) using a writer skill that adapts synthesis depth and source coverage based on mode selection. Standard mode produces quick summaries with key findings; detailed mode includes comprehensive analysis with multiple perspectives; deep mode performs iterative research with multi-agent review-revision cycles. Reports are formatted with markdown, structured sections, citations, and optional image generation. The system uses prompt templates that adapt to research mode and can be customized per deployment.

Implements a multi-agent framework where a ChiefEditor agent orchestrates specialized agents (Researcher, Writer, Reviewer, Reviser) with explicit role definitions and communication protocols. Each agent has specific responsibilities: Researcher gathers information, Writer synthesizes findings, Reviewer validates accuracy, Reviser improves quality. The system uses AG2 (AutoGen) or native orchestration to manage agent state, message passing, and workflow progression. Agents can be configured with different LLM models and parameters to optimize cost and quality per role.

Supports 25+ LLM providers and multiple retriever backends (web search, local documents, vector stores, MCP servers) through a pluggable architecture. The system abstracts retriever interfaces, allowing seamless switching between backends without code changes. Retrievers can be chained or combined (e.g., web search + vector store fallback). Each retriever returns standardized result objects with metadata (source, relevance score, snippet). The configuration system maps retriever selection to research mode and query type, enabling intelligent backend selection.

Integrates with vector stores (Pinecone, Weaviate, Chroma, etc.) for semantic search and retrieval-augmented generation. The system generates embeddings for queries and documents, performs semantic similarity search, and retrieves relevant context from vector stores. Supports configurable embedding models and vector store backends. Results from vector store searches are ranked by relevance score and combined with web search results. The system can use vector stores for both retrieval (finding relevant documents) and context compression (filtering to most relevant chunks).

Loads and parses documents from multiple sources (local files, cloud storage, URLs) in various formats (PDF, DOCX, TXT, Markdown, JSON, CSV, etc.). The system uses format-specific parsers (PyPDF for PDFs, python-docx for Word docs, etc.) and handles extraction of text, metadata, and structure. Supports cloud storage backends (S3, Google Cloud Storage, Azure Blob) for accessing documents without local storage. Parsed documents are converted to standardized internal format with metadata (source, author, date, etc.) for downstream processing.

Abstracts LLM provider interfaces (OpenAI, Anthropic, Ollama, Groq, etc.) through a unified API, supporting 25+ providers. Implements a three-tier model strategy: fast models for planning (e.g., GPT-3.5), standard models for execution (e.g., GPT-4), and advanced models for synthesis (e.g., Claude). Each tier is configurable per deployment, enabling cost optimization by using cheaper models for non-critical tasks. The system handles provider-specific quirks (token limits, function calling formats, rate limits) transparently. Supports local model execution via Ollama for privacy-sensitive deployments.

Implements a FastAPI backend with WebSocket support for real-time research streaming, enabling progressive report updates as research progresses. Clients receive streaming updates for each research stage (query planning, source retrieval, content extraction, synthesis) with intermediate results and progress indicators. The system maintains research state on the server and allows clients to subscribe to specific research tasks. Supports both WebSocket (real-time) and REST API (batch) interfaces for different use cases.