Scrapfly

About Scrapfly

Scrapfly is an API platform for web scraping, browser automation, and structured data extraction. Its core Web Scraping API fetches any URL with anti-bot bypass across 20+ vendors (including Cloudflare, Akamai, DataDome, Imperva, AWS WAF, PerimeterX, F5, and Kasada), proxy rotation across 190+ countries, and JavaScript rendering, returning content as HTML, markdown, or structured JSON. Additional products include a Cloud Browser API for driving remote stealth Chromium sessions over CDP (compatible with Playwright, Puppeteer, and Selenium), a Crawler API for full-site crawls with depth and path controls, a Data Extraction API that converts HTML into structured data using pre-trained models, CSS/XPath templates, or LLM prompts, and a Screenshot API for full-page or viewport captures with diff monitoring.

The platform also offers an AI Browser Agent for natural-language browser automation compatible with tools like Browser Use, Stagehand, and Vibium, and a hosted MCP Server that connects MCP clients such as Claude, Cursor, Cline, and Windsurf to its scraping, screenshot, extraction, and crawl tools. Other components include a CLI for terminal-based scraping, a Proxy Saver for bandwidth optimization with third-party proxy providers, an Unblocker endpoint for anti-bot bypass, and a free Antibot Detector browser extension and API for identifying anti-bot systems on a page. SDKs are available for Python, TypeScript, Golang, Rust, and Scrapy, along with integrations for Zapier, Make, n8n, LlamaIndex, LangChain, and CrewAI.

Scrapfly is aimed at developers and businesses building web scraping, data extraction, or AI agent pipelines, including use cases in e-commerce, real estate, social media, jobs, travel, SEO, and lead generation. Anti-bot bypass is billed on success, with failed attempts not charged. A free tier provides 1,000 credits without a credit card.

Key features

• Web Scraping API with anti-bot bypass on 20+ vendors, proxy rotation, and JS rendering
• Cloud Browser API for remote stealth Chromium sessions over CDP with Playwright/Puppeteer/Selenium support
• Data Extraction API turning raw HTML into structured JSON via pre-trained models, CSS/XPath templates, or LLM prompts
• Screenshot API for full-page, viewport, or element captures in PNG/JPEG/WebP with diff monitoring
• Crawler API for full-site crawls with BFS/DFS depth control and webhook streaming
• MCP Server connecting AI clients like Claude, Cursor, and Windsurf to scrape, screenshot, extract, and crawl

Frequently asked questions