oxylabs-ai-studio-py

Extracts structured data from a single web page using semantic AI understanding rather than CSS selectors or XPath. The AiScraper client sends a URL and natural language prompt to the Oxylabs API, which uses vision and language models to understand page semantics, locate relevant content, and return structured JSON matching the requested schema. This approach is resilient to DOM changes because it operates on semantic meaning rather than brittle selectors.

Discovers and extracts data from multiple related pages across a website using AI-driven navigation. The AiCrawler client accepts a starting URL and a natural language prompt describing which pages to visit (e.g., 'follow all product links and extract prices'), then uses semantic understanding to identify relevant links, navigate to them, and extract data from each page. The SDK manages job polling and pagination internally, returning aggregated results from all discovered pages.

Supports multiple output formats for extracted data, including JSON, HTML, CSV, and raw text. The SDK allows developers to specify desired output format per request, and handles serialization and formatting automatically. This capability enables integration with downstream tools and databases that expect specific formats without requiring post-processing.

Provides comprehensive error handling with detailed diagnostics for extraction failures, including retry logic for transient errors, timeout handling, and structured error messages. The SDK distinguishes between transient errors (network timeouts, temporary API unavailability) and permanent errors (invalid input, authentication failure), applying appropriate retry strategies. Error responses include detailed context (which step failed, why, what was attempted) to aid debugging.

Tracks API usage and enforces rate limits to prevent quota exhaustion. The SDK monitors the number of requests made and remaining quota, and can throttle requests to stay within rate limits. It provides usage statistics and quota warnings to help developers understand their consumption patterns and avoid unexpected quota overages.

Automates complex browser interactions (clicking, form filling, navigation, waiting) using high-level natural language instructions instead of imperative code. The BrowserAgent client accepts a starting URL and an action prompt (e.g., 'log in with email, search for laptops, sort by price'), then uses AI to interpret the prompt, execute the sequence of browser actions, and return the final page state or extracted data. The SDK handles browser session management, JavaScript rendering, and action execution remotely.

Performs web searches and retrieves content from search results using semantic filtering and AI-powered extraction. The AiSearch client accepts a search query and optional filters (e.g., 'find articles about AI safety published in the last month'), then returns a list of search results with extracted content from each page. The SDK handles search engine integration, result ranking, and per-result content extraction internally.

Analyzes a website's structure to discover page hierarchies, relationships, and navigation patterns using semantic understanding. The AiMap client accepts a starting URL and returns a map of the site's structure, including discovered pages, their relationships, and navigation paths. This capability uses AI to understand site semantics (e.g., 'this is a product category page, these are product detail pages') rather than relying on URL patterns or sitemap files.

Enables developers to define JSON schemas that specify the exact structure and types of data to extract from web pages. The SDK accepts a JSON schema (defining fields, types, required properties) and uses it to guide the AI extraction process, ensuring returned data matches the schema structure. This capability works across all extraction clients (AiScraper, AiCrawler, AiSearch) and includes type validation and error handling for schema mismatches.

Manages the asynchronous lifecycle of long-running extraction jobs using a polling pattern that abstracts away HTTP communication details. When a user calls a method like scrape() or crawl(), the SDK submits the job to the Oxylabs API, then polls for completion with exponential backoff, automatic retries on transient failures, and configurable timeouts. The SDK handles all polling logic internally, allowing developers to write synchronous code that blocks until results are ready.

Enables extraction of location-specific content by allowing developers to specify geolocation and language preferences for requests. The SDK accepts geolocation parameters (country, city, IP proxy) and language settings, then routes requests through proxies or renders pages as if accessed from that location. This capability is useful for extracting region-specific pricing, content, or search results that vary by geography.

Automatically renders JavaScript-heavy pages and extracts data from dynamically loaded content. The SDK detects when a page requires JavaScript execution (e.g., React, Vue, Angular apps) and uses a headless browser to render the page, wait for dynamic content to load, and then extract data. This capability is transparent to the user — the SDK handles rendering automatically based on page complexity.

Enables developers to compose complex multi-stage workflows where output from one extraction stage feeds into the next. For example, a developer can first crawl a site to discover product URLs, then scrape each URL to extract detailed data, then search for reviews of those products. The SDK provides utilities to chain operations together, passing data between stages and aggregating results. This capability is useful for building complex data pipelines without writing orchestration code.