API reference
Crawl Site
POST /crawl · Map and crawl
POSThttps://api.clawifi.dev/crawl1 credit per page
- Send your key as
Authorization: Bearer $CLAWIFI_API_KEY; see Authentication. - Add
"dry_run": true(or?dry_run=true) to see what the call would cost; nothing runs and nothing is charged. - Credits: 1 per page (+2 per page that needed JS rendering; at least 1). Every price is on Pricing and credits.
Request#
curl -X POST https://api.clawifi.dev/crawl \
-H "Authorization: Bearer $CLAWIFI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"seed_url":"https://docs.python.org/3/tutorial/"}'from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
result = client.crawl("https://docs.python.org/3/tutorial/")Body
| Field | Type | Default | Required |
|---|---|---|---|
allowed_domains | array of string | no | |
browser_profile | string (optional) | no | |
concurrency | integer | 3 | no |
exclude_patterns | array of string | no | |
force_browser | boolean | false | no |
include_patterns | array of string | no | |
max_depth | integer | 1 | no |
max_links_per_page | integer | 100 | no |
max_pages | integer | 10 | no |
robots_policy | "respect" | "ignore" | "respect" | no |
same_domain_only | boolean | true | no |
scope | "same-origin" | "same-domain" | "subdomains" | "any" (optional) | no | |
seed_url | string | yes | |
timeout_seconds | integer (optional) | no | |
wait_until | "commit" | "domcontentloaded" | "load" | "networkidle" | "domcontentloaded" | no |
Response#
200 with a CrawlResponse object: discovered_count, errors, max_depth_reached, pages, seed_url, success, visited_count.
Errors use the shared shape; see Errors. Try public operations without a key in the playground.
Was this page helpful? Tell us · Last updated 2026-10-22