Diffbot

Automatically extracts structured data from arbitrary web pages without requiring CSS selectors, regex patterns, or manual rules. Uses computer vision to identify and classify page elements (text blocks, tables, images, metadata) and NLP to map them to domain-specific schemas (articles, products, organizations, events, discussions). Processes one page per API call, consuming 1 credit per extraction or 2 credits when routed through datacenter proxies for geo-spoofing or IP rotation.

Crawlbot spiders websites across 50 to 50,000+ URLs, automatically following links and discovering pages within a domain or URL pattern. Applies the Extract API to each crawled page, returning structured data for all discovered pages. Crawling itself consumes zero credits; only the extraction of crawled pages consumes credits (1 per page). Supports configurable crawl depth, URL filtering, and crawl scheduling via the dashboard or API.

Knowledge Graph indexes entities (organizations, articles, products, discussions, events) across multiple languages and regions. Article/News index (1.6B+ records) includes content from global news sources in multiple languages. Organization index (246M+ records) includes companies from multiple regions with localized data (e.g., revenue in local currency, regional employee counts). Product index (3M+ records) includes products from global e-commerce sites. No explicit documentation of supported languages or regions, but scale suggests broad coverage.

Natural Language API extracts named entities (people, organizations, locations, products), relationships between entities (e.g., 'person works at organization'), and topic-level sentiment from raw text documents (1–10,000 characters). Uses NLP models to identify entity types, resolve entity references, and infer relationships without requiring labeled training data or custom entity definitions. Each document consumes 1 credit regardless of length (within the 1–10k character range).

Knowledge Graph API provides query access to Diffbot's pre-indexed database of 10B+ entities across six types: Organizations (246M+ records with 50+ fields), Articles/News (1.6B+ records), Products (3M+ pre-crawled retail products), Discussions (forum/review data with entity matching), Events (23k+ normalized records), and People (scale unknown). Queries use Diffbot Query Language (DQL), a custom SQL-like syntax. Each entity record export consumes 25 credits. Supports filtering, sorting, and aggregation across entity types.

Enhance API enriches existing person or organization records by querying the Knowledge Graph and appending additional fields (revenue, locations, employees, funding, executives for organizations; employment history, education, social profiles for people). Input is a person name/email or organization name/domain; output is enriched record with 50+ fields for organizations or equivalent for people. Each enrichment consumes 1 credit (same as Natural Language API). Integrations available via Excel, Google Sheets, and Zapier for non-technical users.

Diffbot uses a credit-based billing model where each API operation consumes a fixed number of credits: Extract (1 credit), Extract with proxy (2 credits), Natural Language (1 credit), Knowledge Graph export (25 credits), Enhance (1 credit). Monthly plans (Free, Startup, Plus, Enterprise) provide credit allotments at different per-credit rates ($0.001–$0.0009). Overage charges apply at the plan's per-credit rate. Free tier (10,000 credits/month, 5 calls/min) is perpetual with no trial expiration. No long-term contracts required; monthly billing.

Diffbot provides native integrations with Microsoft Excel and Google Sheets, allowing non-technical users to enrich datasets without API integration. Excel integration includes a visual query editor for Knowledge Graph searches and data enrichment. Google Sheets integration supports custom Diffbot Query Language (DQL) formulas for entity lookups and enrichment. Zapier integration enables trigger-based enrichment workflows (e.g., enrich new Salesforce leads with company data). All integrations consume credits at the same rate as direct API calls.

Extract API supports optional datacenter proxy routing, allowing requests to be routed through Diffbot's proxy infrastructure to rotate IP addresses and appear as requests from different geographic locations. Proxy routing consumes 2 credits per extraction instead of 1 credit (100% cost increase). Useful for scraping sites with IP-based rate limiting or geo-blocking. Proxy locations and coverage not documented in source material.

Extract API automatically detects the content type of a web page (article, product, organization, event, discussion) and applies the appropriate extraction schema without user configuration. Each content type has a pre-defined set of fields (e.g., articles: headline, author, publish date, body, images; products: name, price, description, reviews, images; organizations: name, revenue, locations, employees, funding). Detection is based on page structure analysis via computer vision and NLP; no manual content type specification required.

Diffbot enforces rate limits based on pricing tier: Free tier (5 calls/minute), Startup tier (5 calls/second), Plus tier (25 calls/second), Enterprise tier (25+ calls/second, custom). Rate limits apply to all API endpoints (Extract, Natural Language, Knowledge Graph, Enhance). Exceeding rate limits results in HTTP 429 (Too Many Requests) responses. No documented retry logic, backoff strategy, or burst allowance. Rate limits are per-account, not per-API-key or per-endpoint.