[ free tool ]
Is this page agent-readable?
Enter a URL to see the page the way an agent gets it: whether the text is there without JavaScript, how the headings outline it, and how much main text is left once the page furniture is gone.
What it does#
- Reads the page once through Clawifi and checks: readable text came back; the text is there without JavaScript (a plain fetch was enough, no browser needed); no block or challenge page; a title; one H1; an outline of H2s with no skipped levels; enough main text; a published or updated date; links to follow.
- Shows the result as “N of M checks pass”, a plain count with no weighting, plus the heading outline and the extracted text.
- Notes when the page names another address as its canonical one.
Limits#
- Not checked here, because the free preview returns the page’s text and not its HTML: the meta description,
<link rel="alternate">Markdown versions, JSON-LD types, the HTTP status code and robots meta tags. With a key,formats=["raw_html"]returns the HTML to check them yourself. - Headings and length come from the main text Clawifi extracts, so they can differ from the raw HTML.
- The 500-character line for “enough main text” is a rule of thumb, not a standard.
- 12 runs a minute per address, shared with the other free tools.
Do this from your agent#
An agent can run the same read and apply its own checks. Without a key the hosted scrape_url returns Markdown and metadata (12 a minute and 50 a day per address); with a key you can add links and raw_html.
scrape_url(url="https://example.com", formats=["markdown", "metadata", "links"])curl https://api.clawifi.dev/scrape \
-H "Authorization: Bearer $CLAWIFI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","formats":["markdown","metadata","links"]}'from clawifi import Clawifi
client = Clawifi() # reads CLAWIFI_API_KEY
page = client.scrape("https://example.com", formats=["markdown", "metadata", "links"])
print(page["mode_used"], page["metadata"].get("title"))
print([line for line in page["formats"]["markdown"].splitlines() if line.startswith("#")])Read more: Fetch and scrape · POST /scrape · Remote MCP · Docs · Pricing
Questions#
What makes a page readable for AI agents?
Text that is in the HTML the server sends (not only after JavaScript runs), one clear H1, headings that outline the page, a descriptive title, enough main text to quote, visible dates and plain links. This test checks each of those.
Why does JavaScript matter for AI crawlers?
Some agents and AI crawlers read the HTML without running scripts. If the text only appears after JavaScript runs, they may see an empty shell. Server-side rendering or static HTML avoids that.
Is this an SEO score?
No. It is a short checklist with a plain count, and every point is explained. It does not predict rankings or whether any AI product will cite the page.
Why are JSON-LD and the meta description not checked?
The free preview returns the page’s extracted text and metadata, not its HTML, so those tags are not visible here. The API with a key can return the raw HTML for your own checks.
How can I make my pages easier for agents to read?
Render the main text into the HTML, use one H1 and a heading outline, give each page a specific title, show the date it was updated, and consider publishing an llms.txt and Markdown versions of key pages.