ScrapeGraphAI

Converts natural language extraction requirements into directed acyclic graphs (DAGs) of processing nodes without requiring CSS selectors or XPath expressions. The system parses user intent, constructs a node execution plan, and orchestrates LLM calls across a pipeline where each node reads from and writes to a shared state dictionary, enabling declarative scraping workflows that adapt to page structure changes automatically.

Provides a unified abstraction layer supporting 20+ LLM providers (OpenAI, Anthropic, Google, AWS Bedrock, Ollama, Nvidia, etc.) through a common interface, enabling users to swap providers without changing scraping logic. The system handles provider-specific API differences, token counting, model selection, and fallback strategies through a pluggable model registry that maps provider names to concrete LLM implementations.

Processes multi-modal content including images and audio through specialized nodes (ImageToTextNode, TextToSpeechNode) that convert between modalities. Images are converted to text descriptions via vision LLMs, enabling extraction from visual content. Audio is converted to text via speech-to-text, enabling scraping of audio content. This allows scraping workflows to handle rich media content alongside text.

Validates and transforms extracted data against user-defined schemas (JSON Schema, Pydantic models, dataclasses) to ensure output conforms to expected structure and types. The system uses schema_transform utilities to map LLM outputs to typed structures, handle type coercion, and validate constraints. This ensures downstream systems receive data in the expected format with type safety.

Enables fine-grained control over LLM behavior through prompt templates, system messages, and configuration parameters (temperature, max_tokens, top_p, etc.). Users can customize extraction logic by modifying prompts without changing code, and the system supports prompt versioning and A/B testing. This allows optimization of extraction accuracy and cost without modifying graph structure.

Provides mechanisms for handling extraction failures through fallback nodes, retry logic, and error recovery strategies. When a node fails (e.g., LLM call times out, page fetch fails), the system can automatically retry with different parameters, fall back to alternative extraction methods, or skip the node and continue with partial results. This improves robustness for large-scale scraping where some failures are inevitable.

Abstracts web page fetching across four distinct backends (Playwright, Selenium, BrowserBase, Scrape.do) through a unified FetchNode interface, enabling users to choose between local browser automation, cloud-based rendering, or headless scraping based on target site requirements. The system handles JavaScript execution, dynamic content loading, and anti-bot detection transparently, with automatic fallback between backends if configured.

Processes multiple document formats (HTML, PDF, CSV, JSON, XML, Markdown) through a unified parsing pipeline that extracts structured content regardless of source format. The system uses format-specific parsers (HTML via BeautifulSoup/lxml, PDF via PyPDF2/pdfplumber, CSV via pandas, etc.) and normalizes output to a common intermediate representation that downstream LLM nodes can process uniformly.

Provides a BaseNode abstraction that enables developers to create custom processing nodes by implementing a simple interface (execute method that reads from and writes to shared state). Nodes are composable building blocks that can be chained in custom graph topologies, with built-in nodes covering fetch, parse, generate, RAG, search, and conditional logic. The system handles node dependency resolution, state threading, and error propagation automatically.

Integrates Retrieval-Augmented Generation (RAG) capabilities through a RAGNode that embeds document chunks into vector stores (Chroma, Pinecone, Weaviate, etc.) and retrieves relevant context before LLM-based extraction. This enables semantic search over scraped content, reducing token usage and improving accuracy for large documents by providing only relevant excerpts to the LLM rather than full page content.

Integrates web search capabilities through SearchNode and SearchNodeWithContext that query search engines (Google, Bing, DuckDuckGo via SerpAPI, Tavily, etc.) and retrieve results with optional context enrichment. This enables scraping workflows to augment extracted data with real-time search results, perform fact-checking, or gather supplementary information from multiple sources within a single pipeline.

Generates Python code snippets for custom extraction logic based on natural language descriptions and page structure analysis. The system analyzes HTML/document structure, infers extraction patterns, and generates executable Python code that can be executed directly or used as a starting point for further customization. This bridges the gap between declarative natural language requests and imperative extraction code.

Implements conditional branching through ConditionalNode that evaluates conditions on extracted data and routes execution to different downstream nodes based on results. This enables dynamic pipeline behavior where extraction logic adapts based on intermediate results, enabling workflows like 'if price > threshold, extract additional details' or 'if element exists, parse it; otherwise, use fallback'.

Enables batch processing of multiple URLs or documents through graph iteration patterns that apply the same extraction logic across collections of sources. The system handles batching, parallelization (where supported), and result aggregation, allowing users to scrape hundreds of pages with a single graph definition. Multi-source scraping combines results from different sources (web pages, APIs, documents) into unified output.