Web tools
Map and crawl
Map a site from its robots.txt, sitemaps, feeds and links, then crawl it: one bounded call, or a crawl you start, poll, stop and resume.
POST /map#
Map Site. Full parameters on the endpoint page.
curl -X POST https://api.clawifi.dev/map \
-H "Authorization: Bearer $CLAWIFI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
result = client.map("https://example.com")map_site(url="https://example.com")POST /crawl#
Crawl Site. Full parameters on the endpoint page.
curl -X POST https://api.clawifi.dev/crawl \
-H "Authorization: Bearer $CLAWIFI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"seed_url":"https://docs.python.org/3/tutorial/"}'from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
result = client.crawl("https://docs.python.org/3/tutorial/")POST /crawls#
Start Crawl. Full parameters on the endpoint page.
curl -X POST https://api.clawifi.dev/crawls \
-H "Authorization: Bearer $CLAWIFI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"seed_url":"https://docs.python.org/3/tutorial/"}'from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
result = client.post("/crawls", json={"seed_url":"https://docs.python.org/3/tutorial/"})start_crawl(seed_url="https://docs.python.org/3/tutorial/")GET /crawls/{crawl_id}#
Get Crawl. Full parameters on the endpoint page.
curl https://api.clawifi.dev/crawls/<crawl_id> \
-H "Authorization: Bearer $CLAWIFI_API_KEY"from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
result = client.get_crawl("<crawl_id>")get_crawl()POST /crawls/{crawl_id}/stop#
Stop Crawl. Full parameters on the endpoint page.
curl -X POST https://api.clawifi.dev/crawls/<crawl_id>/stop \
-H "Authorization: Bearer $CLAWIFI_API_KEY"from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
result = client.post("/crawls/<crawl_id>/stop")POST /crawls/{crawl_id}/resume#
Resume Crawl. Full parameters on the endpoint page.
curl -X POST https://api.clawifi.dev/crawls/<crawl_id>/resume \
-H "Authorization: Bearer $CLAWIFI_API_KEY"from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
result = client.post("/crawls/<crawl_id>/resume")DELETE /crawls/{crawl_id}#
Delete Crawl. Full parameters on the endpoint page.
curl -X DELETE https://api.clawifi.dev/crawls/<crawl_id> \
-H "Authorization: Bearer $CLAWIFI_API_KEY"from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
result = client.request("DELETE", "/crawls/<crawl_id>")Was this page helpful? Tell us · Last updated 2026-10-22