POST/api/crawl

Submit a Crawl

Submit a new web crawl job. Returns a jobId immediately — the crawl runs asynchronously. Use the job ID to poll for status or configure a webhook for completion notification.

Request

Example requestapplication/json

{
  "url": "https://docs.acme.com",
  "limit": 50,
  "depth": 3,
  "source": "all",
  "formats": ["markdown", "html"],
  "render": false,
  "webhookUrl": "https://yourapp.com/webhooks/crawl"
}

Parameters

Parameter	Type	Description
`url`req	`string`	The starting URL to crawl. Must be a valid http/https URL.
`limit`	`number`	Maximum pages to crawl. Default: 10. Max: 100,000.
`depth`	`number`	Maximum link depth from the starting URL.
`source`	`"all" \| "sitemaps" \| "links"`	Page discovery strategy. Default: "all".
`formats`	`string[]`	Output formats to include: "markdown", "html", "json". Default: ["markdown"].
`render`	`boolean`	Render JavaScript before extracting content. Slower but required for SPAs.
`webhookUrl`	`string`	URL to POST results to when the crawl completes.
`jsonOptions`	`object`	AI extraction config. Requires "json" in formats. See JSON Extraction guide.
`options.includeExternalLinks`	`boolean`	Follow links to external domains.
`options.includeSubdomains`	`boolean`	Follow links to subdomains of the start URL.
`options.includePatterns`	`string[]`	Only crawl URLs matching these glob patterns.
`options.excludePatterns`	`string[]`	Skip URLs matching these glob patterns.

Response

Success response201 Created

{
  "success": true,
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "cfJobId": "cf_job_abc123"
}

400

Bad RequestInvalid parameters or blocked URL (private IP, invalid protocol).

401

UnauthorizedMissing or invalid API key.

402

Payment RequiredInsufficient credits. Upgrade your plan.

502

Bad GatewayCrawl submission to upstream service failed.

Get Job Status JSON Extraction Webhooks