[ free tool ]
robots.txt and llms.txt checker
Enter a domain to see which AI and search crawlers its robots.txt lets in, which sitemaps it lists, and whether its llms.txt has the structure the llms.txt spec describes.
What it does#
- Reads
https://<domain>/robots.txtandhttps://<domain>/llms.txtthrough Clawifi; your browser never contacts the site. - For GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot and Applebot: picks the group that applies (the bot’s own, else
*) and decides the path with the longest matching rule, as RFC 9309 describes. - Lists the
Sitemap:lines, and checks llms.txt for an H1, a blockquote summary, H2 sections and link lists in the- [name](url): noteform.
Limits#
- Two runs per check, of the 12 a minute per address the free tools share.
- It reports what the file says, not what a bot does: robots.txt is a request crawlers choose to honour, and some bots use other tokens or ignore it.
- Bots are matched by their exact token; prefix and vendor-specific matching rules are not applied, and
Crawl-delayis ignored. - It checks llms.txt structure, not whether the linked pages exist. Files over 20,000 characters are cut.
Do this from your agent#
From an agent it is two reads and a little parsing. scrape_url works on the hosted MCP server without a key as a trial (12 a minute and 50 a day per address).
scrape_url(url="https://example.com/robots.txt", formats=["markdown"])
scrape_url(url="https://example.com/llms.txt", formats=["markdown"])curl https://api.clawifi.dev/scrape \
-H "Authorization: Bearer $CLAWIFI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/robots.txt","formats":["markdown"]}'from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
for path in ("/robots.txt", "/llms.txt"):
page = client.scrape("https://example.com" + path, formats=["markdown"])
print(path, page["formats"].get("markdown", "")[:500])Read more: Fetch and scrape · Remote MCP · Docs · Pricing
Questions#
How do I check if GPTBot or ClaudeBot can crawl my site?
Enter the domain above. The table shows, for each AI and search bot, whether its robots.txt group allows the path and which rule decided it.
What is the difference between ClaudeBot and Claude-SearchBot, or GPTBot and OAI-SearchBot?
Each vendor publishes separate tokens for separate uses: GPTBot and ClaudeBot are described as crawling for model training, OAI-SearchBot and Claude-SearchBot as crawling for search results. A robots.txt can allow one and block the other.
What is Google-Extended?
A robots.txt token Google reads to decide whether content may be used for its AI models. It is not a separate crawler: Googlebot still fetches the pages, so blocking Google-Extended does not affect Google Search.
What should an llms.txt file contain?
Per the llms.txt spec: an H1 with the site or project name, a short blockquote summary, optional text, then H2 sections that list links as - [name](url): note. A section called Optional marks links an agent can skip.
Is llms.txt required?
No. It is an optional proposal that some sites publish to give language models a short map of their content. Search engines do not require it.
Does a Disallow rule stop a bot?
Only if the bot honours it. robots.txt is a request, not access control; the reputable crawlers listed here say they follow it.