# robots.txt and llms.txt checker

> Enter a domain to see which AI and search crawlers its robots.txt lets in, which sitemaps it lists, and whether its llms.txt has the structure the llms.txt spec describes.

Canonical page: https://clawifi.dev/tools/llms-txt-checker

Run it on the page: enter a domain and the tool reads its robots.txt and llms.txt and reports what each bot may crawl. No key needed: https://clawifi.dev/tools/llms-txt-checker

## What it does

- Reads `https://<domain>/robots.txt` and `https://<domain>/llms.txt` through Clawifi; your browser never contacts the site.
- For GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot and Applebot: picks the group that applies (the bot’s own, else `*`) and decides the path with the longest matching rule, as RFC 9309 describes.
- Lists the `Sitemap:` lines, and checks llms.txt for an H1, a blockquote summary, H2 sections and link lists in the `- [name](url): note` form.

## Limits

- Two runs per check, of the 12 a minute per address the free tools share.
- It reports what the file says, not what a bot does: robots.txt is a request crawlers choose to honour, and some bots use other tokens or ignore it.
- Bots are matched by their exact token; prefix and vendor-specific matching rules are not applied, and `Crawl-delay` is ignored.
- It checks llms.txt structure, not whether the linked pages exist. Files over 20,000 characters are cut.

## Do this from your agent

From an agent it is two reads and a little parsing. `scrape_url` works on the hosted MCP server without a key as a trial (12 a minute and 50 a day per address).

**MCP**

```text
scrape_url(url="https://example.com/robots.txt", formats=["markdown"])
scrape_url(url="https://example.com/llms.txt", formats=["markdown"])
```

**cURL**

```sh
curl https://api.clawifi.dev/scrape \
  -H "Authorization: Bearer $CLAWIFI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/robots.txt","formats":["markdown"]}'
```

**Python**

```python
from clawifi import Clawifi

client = Clawifi()  # reads CLAWIFI_API_KEY
for path in ("/robots.txt", "/llms.txt"):
    page = client.scrape("https://example.com" + path, formats=["markdown"])
    print(path, page["formats"].get("markdown", "")[:500])
```

Read more: [Fetch and scrape](https://clawifi.dev/docs/fetch) · [Remote MCP](https://clawifi.dev/docs/remote-mcp) · [Docs](https://clawifi.dev/docs) · [Pricing](https://clawifi.dev/pricing)

## Questions

### How do I check if GPTBot or ClaudeBot can crawl my site?

Enter the domain above. The table shows, for each AI and search bot, whether its robots.txt group allows the path and which rule decided it.

### What is the difference between ClaudeBot and Claude-SearchBot, or GPTBot and OAI-SearchBot?

Each vendor publishes separate tokens for separate uses: GPTBot and ClaudeBot are described as crawling for model training, OAI-SearchBot and Claude-SearchBot as crawling for search results. A robots.txt can allow one and block the other.

### What is Google-Extended?

A robots.txt token Google reads to decide whether content may be used for its AI models. It is not a separate crawler: Googlebot still fetches the pages, so blocking Google-Extended does not affect Google Search.

### What should an llms.txt file contain?

Per the llms.txt spec: an H1 with the site or project name, a short blockquote summary, optional text, then H2 sections that list links as `- [name](url): note`. A section called Optional marks links an agent can skip.

### Is llms.txt required?

No. It is an optional proposal that some sites publish to give language models a short map of their content. Search engines do not require it.

### Does a Disallow rule stop a bot?

Only if the bot honours it. robots.txt is a request, not access control; the reputable crawlers listed here say they follow it.

## Other free tools

- [URL to Markdown](https://clawifi.dev/tools/url-to-markdown): A public page as clean Markdown, ready for a prompt.
- [Agent-readability test](https://clawifi.dev/tools/agent-readability-test): Can an agent read this page? Checks, each explained.
- [Sitemap to JSON](https://clawifi.dev/tools/sitemap-to-json): A sitemap.xml as JSON: URLs and lastmod, downloadable.
- [All free tools](https://clawifi.dev/free-tools): The estimator, the request builder and every tool.