Firecrawl Batch Scrape
Scrape many URLs in one asynchronous job — submit a list, then poll the job for status and results.
- Submit —
POST /apis/v1/firecrawl/batch-scrapewith the batch body and a requiredIdempotency-Keyheader. On success you get back202 Accepted, aLocationheader pointing at the job resource (/apis/v1/firecrawl/batch-scrape/{jobId}), and a job object whosestatusstarts asqueued. - Poll —
GET /apis/v1/firecrawl/batch-scrape/{jobId}and repeat untilstatusis terminal:completed,failed, orcancelled. Whencompleted, the scraped documents are inoutput. - List —
GET /apis/v1/firecrawl/batch-scrape(optionallimit,status,cursorquery params). - Cancel —
POST /apis/v1/firecrawl/batch-scrape/{jobId}/cancel.
post_firecrawl-crawl.Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Headers
Unique key (1 to 191 characters) that makes the submit idempotent. Re-submitting with the same key returns the original job.
191Body
1 to 1000 unique HTTPS URLs to scrape. PDF URLs are not supported.
1 - 1000 elementsMaximum number of concurrent scrapes (1 to 20).
1 <= x <= 20Return only the main content of each page.
HTML tags/selectors to keep.
50128HTML tags/selectors to drop.
50128Maximum acceptable cache age in milliseconds.
0 <= x <= 31536000000Minimum cache age in milliseconds before a page is refetched.
0 <= x <= 31536000000Per-page timeout in milliseconds.
1000 <= x <= 300000Response
Batch scrape job accepted. The Location header points at the job resource; poll it until terminal.
An asynchronous integration job. Returned by the submit call (HTTP 202) and by the poll/detail call. Poll the job by its id until status is a terminal value (completed, failed, or cancelled).
Unique AIsa job identifier. Use it to poll, list, or cancel the job.
"iaj_01HZY8Q2M4K7N9V3T6W1X0B2C3"
Always "integration_async_job".
"integration_async_job"
The submit endpoint this job belongs to.
"/apis/v1/firecrawl/crawl"
Customer-facing lifecycle status. queued and running are non-terminal; completed, failed, and cancelled are terminal.
queued, running, completed, failed, cancelled When the job was accepted.
When the job reached a terminal status. Null while the job is still queued or running.
Job result payload. Present only once status is completed. For crawl this is the array of scraped pages; for batch scrape it is the array of scraped documents.
True when the result has been retained past its retention window and is no longer retrievable.
Present when status is failed. Null otherwise.