TL;DR
- A search API answers "which pages match this query?" A scraping API answers "what's on this page?"
- If you start from a question, you need search. If you start from a URL list, you need scraping. Most AI apps need both.
- Scraping APIs bill per page (Firecrawl: 1 credit per page). Search APIs bill per query ($0.03–25 per 1,000).
- Search + content APIs do both in one call: Exa, Tavily, Firecrawl Search and Keirolabs (from $1.33 per 1,000 with page markdown).
I’ve built scrapers for fifteen years, and the most common mistake I see isn’t a bad parser. It’s someone using a scraping tool for a search problem, or a search tool for a scraping problem, then wondering why the bill is huge or the data is thin.
The two get lumped together because they both “get stuff from the web”. They do different jobs.
The one-line difference
- A search API takes a query and returns pages that match: URLs, titles, snippets.
- A scraping API takes a URL and returns what’s on that page: HTML, clean text, markdown or structured fields.
Search finds. Scraping reads.
| Search API | Scraping API | |
|---|---|---|
| Input | A query (“best crm for startups”) | A URL (or a list of them) |
| Output | Ranked list of matching pages | The content of one page |
| Billed per | Query | Page (sometimes per GB or per render) |
| Hard part | Having a good index | Getting past blocks, rendering JavaScript |
| Examples | Brave, Serper, Exa | Firecrawl, Jina Reader, ScraperAPI |
When you need a search API
You need search when you don’t know the URLs yet:
- An AI agent answering questions about current events
- A research tool that finds sources on a topic
- Rank tracking (“where does my page show up for this keyword?”)
- Lead generation from a query (“dentists in Austin”)
When you need a scraping API
You need scraping when you already have the URLs:
- Monitoring competitor prices on known product pages
- Turning your docs site into markdown for a RAG index
- Pulling details from a list of company websites
- Archiving pages
The hard part of scraping is everything around the fetch: rotating proxies, rendering JavaScript, solving or avoiding bot checks, retrying. Scraping APIs exist so you don’t run that infrastructure yourself.
Scraping a page is not the same as having the right to use it. Check the site’s terms, respect robots.txt where it applies, and get legal advice for anything commercial at scale. Search APIs with their own index avoid most of this, because they’ve done the crawling under their own policies.
When you need both (most AI apps)
An agent that answers “what changed in the EU AI Act this month?” needs to find the right pages, then read them. The classic stack was two vendors:
- Call a search API → get 10 URLs
- Call a scraping API on each → get 10 pages of text
- Feed the text to the model
That’s 11 calls, two bills and two failure modes per question. It’s also slow, because step 2 waits on ten page loads.
APIs that search and read in one call
The newer option: one API that returns search results with page content.
| API | Search + content per 1,000 | Notes |
|---|---|---|
| Keirolabs | $1.33–4.00 (search/content, 1.5 credits) | Own index, markdown for RAG, plus JS-rendering extract for hard pages |
| Exa | $4–7 search + $1 per 1,000 pages | Own neural index |
| Tavily | $5–8 basic | Cleaned content in results |
| Firecrawl Search | 2 credits per 10 results + scrape | Search and scrape in one call |
| Parallel | $1–5 | Ranked, compressed excerpts rather than full pages |
If you’re starting fresh, start with a search + content API and only add a separate scraper for the pages it can’t read. Keirolabs is the cheapest way I know to cover both halves: search with page markdown from $1.33 per 1,000 on Startup, and a URL-to-markdown extract at 2 credits that steps up to JavaScript rendering for the stubborn sites. That used to take two or three separate tools.
What each costs, side by side
For 100,000 questions a month, each needing search plus the text of the top 5 pages:
| Approach | Rough monthly cost | Calls per question |
|---|---|---|
| Brave search ($5) + Firecrawl Growth (500,000 pages) | $500 + $399 = $899 | 6 |
| Exa auto ($7) + contents ($1 per 1,000 pages × 5) | $700 + $500 = $1,200 | 1 |
| Keirolabs search/content (1.5 credits) | 150,000 credits: Startup’s 112,500 ($99.99) + two $100 packs (40,000) ≈ $300 | 1 |
These are list prices; the right answer depends on how many pages you really need per question. Run your own numbers in the calculator.
How to choose
- Do you start from questions or URLs? Questions → search. URLs → scraping.
- Does a model read the output? Then get page text in the same call.
- Are the target sites hard? Heavy JavaScript and bot walls need a real browser. Check that your provider has one.
- Count calls per task, not just price per call. One call that returns text beats eleven that don’t.
FAQ
Is a SERP API a scraping API?
Sort of. A SERP API scrapes Google’s results page for you and parses it. It’s a scraper aimed at one site, used for search. See what is a SERP API.
Can I just use a scraping API on Google?
You can try, but Google blocks it aggressively, and you’d be rebuilding a SERP API. Use one instead: best SERP APIs.
What’s the cheapest way to get page text for search results?
Among the APIs we track, Keirolabs’ search/content endpoint: 1.5 credits per call, about $1.33 per 1,000 on Startup and $4.00 on Essential. Exa adds $1 per 1,000 pages on top of its search price.