Render JavaScript-heavy pages with headless Chromium and extract Markdown, HTML, links, images, or screenshots. Batch up to 100 URLs or crawl whole sites asynchronously.
This runs the live Web Scraping API against a set of sample pages: headless Chromium renders the page, then Readability extraction returns clean content.
The demo is limited to these sample pages. Production scrapes any public https URL.
Production also supports rawHtml, images, and screenshot formats.
Rendering page in headless Chromium...
Why Web Scraping API
Real browser rendering
Headless Chromium executes JavaScript before extraction, so SPAs and dynamic pages return their actual content, not empty shells.
Six output formats
markdown, html, rawHtml, links, images, and screenshot, selectable per request via the formats array.
Main-content extraction
Readability mode strips navigation, ads, and boilerplate by default. Fine-tune with includeTags and excludeTags CSS selectors.
Batch and crawl jobs
Scrape up to 100 URLs per async batch, crawl entire sites with depth and path filters, or map a site's URLs synchronously.
Browser actions
Chain up to 50 pre-extraction actions like clicks, scrolls, and waits to reach content behind interaction.
SSRF-guarded by design
DNS-validated guards block private, link-local, CGNAT, and cloud-metadata hosts on the entry URL, redirects, and subresources.
/api/v1/scrape
Synchronous scrape
Single-URL scrape returning content in the requested formats. 35-second route timeout.
Start batch job
Async scrape of 1 to 100 URLs. Returns 202 with a job id to poll.
Poll batch job
Job status with completed and failed counts plus per-URL results. Paginate with ?skip=N.
Start site crawl
Async crawl from a seed URL with limit, maxDepth, includePaths, and excludePaths.
Poll crawl job
Crawl progress and page results. DELETE the same path to cancel a running crawl.
URL discovery
Synchronous site map: returns the links reachable from a URL. 60-second timeout.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | yes | Public https URL to scrape. SSRF-guarded on entry and redirects. |
| formats | array | no | Output formats, default ["markdown"]: markdown, html, rawHtml, links, images, screenshot. |
| onlyMainContent | boolean | no | Apply Readability extraction. Default true. |
| waitUntil / waitFor | string / integer | no | Navigation wait strategy (load, domcontentloaded, networkidle0/2) plus up to 10,000 ms of extra wait. |
| includeTags / excludeTags | array | no | CSS selectors to keep or remove before extraction. |
| actions | array | no | Up to 50 browser actions (click, scroll, wait) executed before extraction. |
| maxAge | integer | no | Redis cache TTL in ms for repeat scrapes. 0 (default) bypasses the cache. |
| urls | array | batch only | 1 to 100 URLs for POST /batch/scrape. |
Parameters shown are for POST /scrape; batch, crawl, and map accept the same scrape options where applicable.
Error reference
Scraping fails in predictable ways. Every error includes a correlationId and enough detail to decide between retrying and rerouting.
Malformed request body, or the URL resolves to a private, link-local, CGNAT, or cloud-metadata host.
Missing or invalid API key. Send x-api-key or api_key with every request.
Unknown endpoint, or a batch/crawl job id that does not exist or has expired.
The page loaded but no extractable content was found. Try rawHtml or adjust tag filters.
Request limit exceeded or the browser pool is at capacity. Retry with backoff.
The page exceeded the route budget: 35 s for /scrape, 60 s for /map. Use batch jobs for slow sites.
Example error response
{
"success": false,
"error": "URL blocked by SSRF guard",
"correlationId": "req_abc123"
}FAQ
Yes. Every request renders the page in headless Chromium before extraction, so single-page apps and dynamically loaded content return their real markup. Control timing with waitUntil and waitFor, and interact first using the actions pipeline.
Six: markdown (default), html, rawHtml, links, images, and screenshot. Request several at once with the formats array, e.g. ["markdown", "links"].
POST /batch/scrape accepts up to 100 URLs and returns a job id to poll. POST /crawl walks a site from a seed URL with limit, maxDepth, and path filters. POST /map returns a site's URLs synchronously.
The service enforces a DNS-validated SSRF guard: private ranges, link-local, CGNAT (100.64.0.0/10), and cloud-metadata hosts are blocked on the entry URL, every redirect, and page subresources. Only public https URLs are accepted.
A typical scrape completes in a few seconds; the route budget is 35 seconds. Set maxAge in milliseconds to serve repeat scrapes from Redis instead of re-rendering.
Markdown for LLMs, links for crawlers, screenshots for archives. One API handles rendering, extraction, batching, and the security guardrails.