Website Structure Discovery And Url Mapping

Scrape websites and extract structured data via Firecrawl MCP.

Unique: Provides lightweight URL discovery without content extraction, allowing agents to plan scraping strategy before committing credits to full content fetches. The depth-based crawling with pattern filtering enables selective discovery — agents can discover only URLs matching specific criteria (e.g., /blog/* paths) without exploring entire site.

vs others: More efficient than scraping every page to build a sitemap because it skips content extraction; more reliable than parsing robots.txt or sitemaps.xml because it performs actual crawling and discovers dynamically-linked content.