A web scraping API takes a URL and returns the page in a form a model can read, usually Markdown or JSON. The providers differ in what they do when a page needs JavaScript, blocks automated traffic, or is one of ten thousand pages on a site. Details below were checked against each provider's official pages in September 2026.
What is the best web scraping API for AI agents?
For most agents, start with Firecrawl: it returns Markdown or schema-based JSON, crawls whole sites, and has a hosted MCP server. Use Jina Reader for the simplest way to read one URL, Bright Data or Zyte when sites block you, Apify for site-specific scrapers, Crawl4AI to self-host, and Tavily Extract when you already use Tavily for search.
| API | Consider it when you need… | What to evaluate |
|---|---|---|
| Firecrawl | Markdown, structured JSON, and site crawls from one API | Success on your protected targets, and the extra credits for JSON extraction |
| Jina Reader | To read a known URL with almost no setup | Extraction quality on your pages and token-based cost at volume |
| Apify | Ready-made scrapers for specific sites | Quality and cost of the specific Actors you depend on |
| Bright Data | Pages behind anti-bot systems and CAPTCHAs | Per-request cost against how often simpler tools fail on your targets |
| Zyte | Per-request pricing that follows site difficulty, with typed extraction | Which price tier your target sites fall into |
| Crawl4AI | An open-source crawler you run yourself | The operating work: browsers, proxies, and scaling |
| Tavily Extract | Page content from URLs your search step already found | Whether basic depth captures tables and embedded content you need |
How do web scraping APIs compare on pricing and features?
The main differences are the billing unit (credits, tokens, requests, or compute), who handles blocking, and whether the provider runs an MCP server for you. All figures come from the linked official pages.
| API | Free tier | Paid pricing | JavaScript rendering | Output formats | Crawl or single page | Anti-bot and proxies | MCP server |
|---|---|---|---|---|---|---|---|
| Firecrawl | 1,000 credits a month, 2 concurrent requests (pricing) | Plans from $16 a month billed annually for 5,000 credits; 1 credit per page, +4 for JSON mode | Yes, plus page actions (scrape docs) | Markdown, HTML, links, screenshot, JSON by schema or prompt, summary | Both; crawl is an async job, 10,000-page default limit | Basic, stealth, and auto proxy modes | Hosted at mcp.firecrawl.dev, with keyless, OAuth, or API-key access (docs) |
| Jina Reader | 20 requests a minute with no key; new keys include free tokens (Reader page) | Billed by output tokens; top-up prices are on the vendor page | Yes, default browser engine; a direct HTTP engine is also available | Markdown, JSON response; schema extraction through ReaderLM-v2 | Single page | Not a stated focus on the pages we checked | Hosted at mcp.jina.ai; read_url works without a key, rate limited (repo) |
| Apify | $5 of usage each month (pricing) | Starter $19, Scale $199, Business $999 a month; compute units from $0.20 | Depends on the Actor | Depends on the Actor; results land in datasets | Both, depending on the Actor | Depends on the Actor | Hosted at mcp.apify.com with OAuth or token (docs) |
| Bright Data | 5,000 requests a month through its MCP server (docs) | $1.50 per 1,000 requests pay-as-you-go; Scale plan $499 a month (Web Unlocker pricing) | Yes; remote browser tools in the MCP server | Markdown, HTML, JSON from site-specific extractors | Single page and site-specific datasets | Proxy rotation, anti-bot challenges, and CAPTCHA solving on every request | Hosted and self-hosted (repo) |
| Zyte | $5 credit for 30 days (pricing) | Per 1,000 requests by site tier: $0.13–$1.27 HTTP, $1.01–$16.08 browser-rendered, pay-as-you-go | Yes, billed at the browser rate | HTML, plus AI extraction into products, articles, job postings, and custom schemas (Zyte API) | Single-request API | Automatic ban handling and IP rotation on all plans | Not listed on the pages we checked |
| Crawl4AI | Free and open source, Apache 2.0 (docs) | Your own infrastructure and any LLM calls | Yes, runs a real browser | Markdown; CSS, XPath, or LLM-based structured extraction | Both; deep crawling is built in | Proxy support, stealth modes, and an undetected browser option; you supply the proxies | Built into the Docker server at /mcp/sse (self-hosting) |
| Tavily Extract | 1,000 credits a month (credits) | $0.008 per credit pay-as-you-go; plans $30–$500 a month. 1 credit per 5 URLs basic, 2 advanced | Not stated; advanced depth retrieves tables and embedded content (Extract reference) | Markdown or text | Up to 20 URLs per request; separate crawl and map endpoints | Not stated on the pages we checked | Hosted at mcp.tavily.com (docs) |
What does an AI agent need from a scraping API?
An agent needs the page content in few tokens, a clear failure when a page could not be read, and structured fields when the result feeds another step. Consider this request:
Every Monday, read the pricing pages of these five competitors. Record each plan name, price, and usage limit, and tell me what changed since last week.
That single request touches four capabilities:
- Rendering: pricing pages often build their tables with JavaScript, so a plain HTTP fetch returns an empty shell.
- Clean output: Markdown without navigation and footers costs fewer tokens than raw HTML.
- Structured extraction: plan, price, and limit as JSON fields make the week-over-week comparison reliable.
- Access: some sites block datacenter traffic. The provider either handles that or the agent gets an error page.
Scraping is different from search. A web search API finds the pages; a scraping API reads them. When the agent has to log in, click, or fill forms, it needs a browser automation tool instead.
Firecrawl
The default choice when you want Markdown, structured JSON, and crawling from one API.
The scrape endpoint returns Markdown, HTML, links, screenshots, a summary, or JSON extracted with a schema or a prompt. It renders JavaScript and can run page actions such as click and wait. Crawl runs as an asynchronous job with a default limit of 10,000 pages, respects robots.txt by default, and can notify a webhook.
Pricing is credit based: one credit per page, four more for JSON mode. The free plan includes 1,000 credits a month. The hosted MCP server supports keyless use within daily limits, OAuth sign-in, or an API key.
Jina Reader
The simplest way to turn one known URL into Markdown.
Put https://r.jina.ai/ in front of a URL and
Reader returns the page as
Markdown. It works without a key at 20 requests a minute, and at 500 a
minute with a free key. The default engine runs JavaScript; ReaderLM-v2
can extract JSON against a schema. Usage is billed by output tokens.
The Jina MCP server exposes
read_url and parallel_read_url, which work
without a key under rate limits. Reader reads single pages; it does not
crawl a site.
Apify
Worth evaluating when your targets are specific sites that already have a maintained scraper.
Apify is a marketplace of scrapers called Actors, each with its own input and output schema. Through the Apify MCP server, an agent can search the store, inspect an Actor, run it, and read the results from a dataset. It connects with OAuth or an API token.
Pricing is usage based. The free plan includes $5 of usage each month, and compute units cost $0.20 on the Free and Starter plans. Rendering, anti-bot handling, and output quality depend on the Actor, so evaluate the specific Actors you plan to rely on.
Bright Data
A candidate when the hard part is getting the page at all.
Bright Data's MCP server runs on its Web Unlocker, which handles proxy rotation, anti-bot challenges, and CAPTCHA solving on every request. Tools return Markdown or HTML for general pages and JSON from site-specific extractors. Tool groups let you load only what the agent needs, which keeps its context small. Remote browser tools are included.
The MCP server includes 5,000 free requests a month. After that, Web Unlocker pricing is $1.50 per 1,000 requests pay-as-you-go, charged for successful requests only. Note that the hosted endpoint takes the API token in its URL, so treat that URL as a secret.
Zyte
An option when you want cost to track how hard each site is to scrape.
Zyte API bundles ban handling, IP rotation, browser rendering, sessions, and page actions in one request. Its AI extraction returns typed data for products, articles, and job postings, plus custom schemas.
Pricing is per 1,000 requests across five site-difficulty tiers: $0.13 to $1.27 for HTTP responses and $1.01 to $16.08 for browser-rendered ones, pay-as-you-go. Monthly commitments from $100 lower those rates. New accounts get a $5 credit for 30 days. We did not find an official MCP server on the pages we checked, so plan to wrap the API as a tool yourself.
Crawl4AI
The open-source option for teams that want to run the crawler themselves.
Crawl4AI is an Apache 2.0 Python library that drives a real browser and produces Markdown, with CSS, XPath, or LLM-based structured extraction. It supports deep crawling, session reuse, proxies, and stealth modes.
The Docker server listens on port 11235 and includes MCP endpoints with tools for Markdown, HTML, screenshots, PDFs, JavaScript execution, and crawling. There is no per-page fee. You pay in operations: browsers, proxies, and scaling are yours to run.
Tavily Extract
A good fit when Tavily already handles search and you need the page content behind the results.
Extract accepts up to 20 URLs per request and returns Markdown or text. Advanced depth retrieves more, including tables and embedded content. The response lists failed URLs separately, so check both arrays.
Credits: one credit per five successful basic extractions, two for advanced. The free tier is 1,000 credits a month and pay-as-you-go is $0.008 per credit. The Tavily MCP server exposes search and extract. Tavily does not return schema-based JSON; your model does that step.
What do the benchmarks say about web scraping APIs?
Published benchmarks agree on one thing: success rates on protected sites vary widely between providers. We have not run our own benchmark. The results below are published by others, and both publishers have commercial ties to the market, so read them as directional.
Scrape.do's 2026 benchmark (vendor-run)
Scrape.do tested 16 scraping APIs against seven protected targets: Amazon, Indeed, GitHub, Zillow, Capterra, Google, and X. It checked that responses contained the expected page elements, not only an HTTP 200. Scrape.do sells a competing API and ranks itself first for value.
| Provider | Reported average success rate |
|---|---|
| Bright Data | 98.87% |
| Apify | 97.14% |
| Firecrawl | 60.47% |
The write-up also notes that Firecrawl declines to scrape some social sites at the API layer, which lowers its score on this target set. These are heavily defended sites. Documentation, blogs, and most company pages are a different workload.
AIMultiple's benchmark
AIMultiple reports 12,500 requests across more than 3,000 URLs in e-commerce, search, and social media. It highlights Bright Data for the range of data returned and Zyte for a median response of about two seconds on Amazon and Target. It does not cover Firecrawl, Jina Reader, Crawl4AI, or Tavily.
Neither benchmark measures what matters most to an agent: whether the Markdown or JSON that comes back is complete and clean. You need to check that on your own pages.
Firecrawl vs Jina Reader vs Bright Data: which should you choose?
Choose by the hardest part of your task:
- Firecrawl for general agent work. One API covers reading a page, extracting JSON, and crawling a site.
- Jina Reader for reading known URLs. No account is needed to start, and there is little to configure.
- Bright Data for blocked sites. Unblocking is part of every request, and you pay for successful ones.
Pick the closest fit, then run one alternative on the same URLs.
How do you test a scraping API on your own task?
Collect 20 to 30 URLs your agent will really read, send the same set to two or three providers, and compare the output by hand. Keep the model and instructions fixed so only the scraper changes.
| Page type | What a useful result looks like |
|---|---|
| Documentation page | Headings, code blocks, and tables intact; no navigation noise |
| JavaScript-rendered pricing page | All plans and prices present, not an empty shell |
| Long article | Full body text without ads, related links, or cookie banners |
| Product or listing page | Correct fields in JSON, with nulls where the page has no value |
| A site that blocks bots | The real page, or a clear error instead of a CAPTCHA page returned as content |
| PDF or other file | Readable text, with the per-page cost understood |
Record the success rate, latency, cost per successful page, and the tokens your model spent reading the output. Then open a few failures. A blocked request, a rendering timeout, and an extraction that dropped a table need different fixes.
How do I give my agent web scraping without managing API keys?
Use a provider's hosted MCP server and connect it to your agent
once. Firecrawl's server supports keyless use within daily limits
and OAuth sign-in, Apify's uses OAuth, and Jina's
read_url works without a key under rate limits.
In Dexto, connections accept any custom MCP server, and Dexto discovers the tools that server offers. Add the scraper's MCP URL in Connections settings, or ask Dexto to connect it for you. Your agent code never handles the credential, and you can remove the connection at any time. Our MCP server shortlist covers other servers worth connecting.
Where does Dexto fit?
Dexto is the agent that uses the scraper. A scraping API returns page content; you still need something to decide which pages to read, keep the results, compare them over time, and deliver the output. That surrounding system is the harness, and it is what Dexto provides.
- Tools: connections include more than 50 built-in integrations and any custom MCP server, so a scraper sits next to Slack, Notion, or GitHub in the same agent.
- A computer: Dexto has a persistent cloud computer where it stores files, runs code, and opens browsers. When an API cannot read a page, the agent can open it in a browser. It can also save last week's scrape and diff it against this week's.
- Process: a skill holds your scraping rules, such as which fields to extract and what to do when a page fails. An agent configured for the job applies them every time.
For the competitor pricing example, that means: connect a scraper, describe the five pages and the fields in a skill, and have the agent write the comparison to a file or post it to Slack.
Is web scraping for AI agents legal?
It depends on the site, the data, and your jurisdiction, and this is not legal advice. As a baseline, read each site's terms, respect robots.txt (Firecrawl's crawler does by default), avoid personal data, and prefer an official API when one exists.
Try it in Dexto
Pick one scraper from the table, connect its MCP server, and give Dexto a real task. Open Dexto or read the quickstart.