Scrapling

Implements a three-tier fetcher system (Fetcher for static HTTP, dynamic browser fetcher for JavaScript-heavy sites, StealthyFetcher for anti-bot detection) where all tiers return the same Response object inheriting from Selector. This allows developers to start with fast HTTP requests and transparently upgrade to browser automation without changing parsing code. Uses lazy imports via __getattr__ to defer loading heavy dependencies (Playwright, browser engines) until first access, minimizing initial memory footprint and import latency.

Implements intelligent selector resolution that automatically relocates elements when DOM structure changes between requests, using tree-sitter AST parsing or similar structural analysis to maintain selector validity across page mutations. When a CSS or XPath selector fails, the system analyzes the current DOM and attempts to find the target element using fallback strategies (attribute matching, structural similarity, text content matching). This enables robust scraping of pages with dynamic or inconsistent HTML structures without manual selector maintenance.

Provides extensible middleware system for transforming requests and responses through custom handlers. Developers can register custom type handlers that convert Response objects to domain-specific types (e.g., JSON, CSV, custom dataclasses) or apply transformations (e.g., text cleaning, data validation). Middleware is applied in a pipeline: request → fetcher → response → handlers → output. Handlers can be conditional (applied only to certain URLs or response types) and composable (chained together). The system supports both synchronous and asynchronous handlers for integration with async crawlers.

Provides command-line interface (CLI) and interactive REPL shell for testing scrapers without writing code. The CLI supports common operations (fetch URL, parse HTML, extract data) with flags for fetcher selection, proxy configuration, and wait strategies. The interactive shell allows developers to iteratively test selectors, refine extraction logic, and debug issues in real-time. Shell sessions maintain state (current URL, parsed HTML, session cookies) across commands, enabling rapid iteration. Output can be formatted as JSON, CSV, or pretty-printed for easy inspection.

Implements lazy loading of heavy dependencies (Playwright, browser engines, proxy libraries) through __getattr__ dynamic imports, reducing initial import time and memory footprint. The system provides resource pooling for browser instances and HTTP connections, automatic cleanup of unused resources, and memory-efficient DOM parsing using streaming where possible. Configuration options allow tuning of pool sizes, timeouts, and resource limits. Monitoring hooks expose resource usage metrics (active connections, browser tabs, memory) for performance analysis and optimization.

Provides StealthyFetcher class that configures Playwright with anti-bot detection evasion techniques including: disabling headless mode indicators, spoofing user agents and device properties, managing WebDriver detection flags, implementing realistic mouse/keyboard behavior patterns, and rotating proxy/IP addresses. The system integrates with proxy rotation middleware to distribute requests across multiple IPs, and configures browser launch parameters to minimize detection signatures. All evasion techniques are composable and can be selectively enabled based on target site requirements.

Implements Selector class that wraps BeautifulSoup4/lxml and provides unified API for both CSS and XPath selectors, returning Response objects that themselves inherit from Selector for chainable query syntax. Supports advanced selector features including pseudo-selectors, attribute matching, text content filtering, and relative selectors. The Response object maintains context about the source (HTTP, browser, stealth) and allows seamless chaining of selectors (e.g., response.css('div.item').xpath('.//span[@class="price"]').text()).

Provides Session and AsyncSession classes that manage connection pooling for HTTP requests and browser tab pooling for Playwright-based fetchers. HTTP sessions reuse TCP connections to reduce latency and overhead. Browser sessions maintain a pool of tabs (configurable size) that are recycled across requests, avoiding the overhead of launching new browser instances. Sessions also manage cookies, headers, and authentication state across multiple requests, with optional persistence to disk. The architecture supports concurrent request handling through async/await patterns.

Provides Spider base class that enables declarative crawl patterns through method overrides (start_requests, parse, parse_item) and lifecycle hooks (on_request_start, on_response_received, on_error). Spiders define crawl logic by overriding parse() to extract data and yield new requests, creating a declarative crawl graph. The framework handles request queuing, deduplication, and response routing automatically. Spiders integrate with sessions for connection pooling and support custom middleware for request/response transformation. The architecture follows Scrapy's proven Spider pattern but with Scrapling's unified Response interface.

Implements proxy rotation middleware that distributes requests across a configured proxy pool (residential, datacenter, or custom proxies) with automatic fallback when proxies fail. Supports proxy authentication (username/password), per-request proxy selection, and rotation strategies (round-robin, random, weighted). Failed proxies are temporarily blacklisted and retried after a cooldown period. The system integrates with both HTTP fetchers (via httpx proxy config) and browser fetchers (via Playwright proxy settings). Proxy state is tracked across requests and can be persisted for resuming crawls.

Provides configurable wait strategies for browser-based fetchers to handle dynamic content loading: wait for specific elements (CSS/XPath), wait for network idle, wait for JavaScript execution completion, or custom wait conditions. The system detects when a page has finished loading by monitoring network activity, DOM mutations, and JavaScript execution state. Wait strategies are composable (e.g., wait for element AND network idle) and can be applied per-request or per-session. Timeout handling ensures requests don't hang indefinitely on slow or broken pages.

Exposes Scrapling as a Model Context Protocol (MCP) server, allowing AI agents and LLMs to invoke scraping operations through a standardized tool interface. The MCP server wraps Scrapling's fetchers, spiders, and selectors as callable tools with schema-based function signatures. AI agents can compose scraping workflows by chaining tool calls (e.g., fetch URL → parse HTML → extract data → follow links). The server handles tool invocation, error handling, and response serialization transparently. Integration with Claude, ChatGPT, or custom LLM agents enables natural language scraping instructions to be translated into Scrapling operations.

Implements async-first architecture using Python asyncio for concurrent request handling, with built-in request queuing (FIFO or priority-based) and automatic URL deduplication using bloom filters or in-memory sets. The engine manages concurrent request limits (configurable per-domain or global) to respect rate limits and avoid overwhelming target servers. Failed requests are automatically retried with exponential backoff. The system tracks crawl statistics (requests sent, responses received, errors, deduplication hits) for monitoring and debugging. Distributed crawling is supported through external task queues (Celery, RQ) for multi-process/multi-machine scaling.