Help
Frequently asked questions
Everything you need to know about AI crawlability and how the tool works.
What is AI crawlability?
AI crawlability is how well AI systems — like ChatGPT, Claude, and Perplexity — can discover, access, and understand your website's content. A crawlable site has the right files (llms.txt, robots.txt, sitemap.xml), structured data, clean HTML, and public pages AI agents can index.
What is llms.txt?
llms.txt is a Markdown file placed at /llms.txt on your site. It gives AI agents a structured overview of your site: what it does, what pages exist, and where key content lives. Think of it as a table of contents written for AI. The spec is documented at llmstxt.org.
How is AI crawlability different from regular SEO?
Traditional SEO focuses on signals like backlinks, keyword density, and page speed for Google and Bing. AI crawlability focuses on machine-readable structure — llms.txt, JSON-LD schema, semantic HTML, clean text, and explicit crawl permissions for AI bots — so AI systems can understand and cite your content accurately.
Which AI crawlers should I allow in robots.txt?
The most important ones are GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), ChatGPT-User, and PerplexityBot. Add explicit User-agent sections with Allow: / for each. Never block them unless you have a legal reason to.
What is structured data and why does it matter for AI?
Structured data is JSON-LD markup embedded in your pages that describes what your content is — Article, Product, FAQPage, Organization, etc. AI systems use it to understand page context without reading every word, which makes your content more likely to appear in AI-generated answers.
How does the crawl test work?
When you enter a URL, our tool fetches your site in real time using our own crawler bot and runs 20+ checks. Every result is live — nothing is cached. We check for llms.txt, robots.txt directives, sitemap.xml, structured data, Open Graph tags, HTML structure, internal linking, authority signals, and more.
What score should I aim for?
A grade of A (90+) means your site follows most best practices. A B (75+) is good but has room to improve. Anything below C (60) means there are significant gaps that could be hurting your visibility in AI-powered search results.
Is the tool free?
Yes, the full audit is free. You can see the score summary and first few checks immediately. Unlock all checks and recommendations by entering your email — still free, no payment required.
How often should I re-check my site?
Re-run the audit whenever you make significant changes to your site structure, metadata, or content strategy. We also recommend checking quarterly as AI crawler requirements evolve.
What is an MCP server card?
An MCP (Model Context Protocol) server card is a JSON file at /.well-known/mcp.json that describes what tools or capabilities your site exposes to AI agents. It lets agents discover and use your site's functionality automatically, similar to how OpenAPI specs describe REST APIs.
What is a crawl test?
A crawl test is a live check that fetches your website the same way a search or AI crawler would, then reports what it can and cannot access. A good crawl test covers robots.txt directives, sitemap availability, page response codes, structured data, and AI-specific files like llms.txt. Our free website crawlability tester runs all of these checks in real time — no caching, no signup.
How do I check website crawlability?
To check crawlability, enter your domain into this tool and click 'Check site'. Within seconds you get a full crawlability audit across 20+ signals: AI crawler permissions, file presence, structured data, heading structure, internal linking, authority signals, and more. Each failing check comes with a specific fix recommendation.
What does a crawlability checker test?
A crawlability checker verifies every layer that affects whether bots can index your site. This includes: (1) robots.txt — are AI user-agents explicitly allowed? (2) sitemap.xml — is it present and linked? (3) llms.txt — does it exist and is it valid? (4) Structured data — is JSON-LD markup present? (5) Page speed and compression — does your server respond quickly? (6) Meta tags and Open Graph — are pages described correctly? Our crawlability checker tests all of these and more.
Is there a free online crawlability checker?
Yes. This tool is a completely free website crawlability checker — no account required, no payment, and no rate limit on single checks. Enter any URL to instantly run a full crawling test and get a scored report. You unlock all check details and recommendations by providing your email, which is still free.
What is a crawling test for AI search engines?
A crawling test for AI search engines goes beyond standard SEO crawl tools. It specifically verifies whether AI crawlers like GPTBot, ClaudeBot, and PerplexityBot are permitted in robots.txt, whether llms.txt is present and correctly formatted, and whether your structured data gives AI agents enough context to cite your content accurately in AI-generated answers.
How is a website crawl checker different from Google Search Console?
Google Search Console shows you historical crawl data from Googlebot only. A website crawl checker like ours runs live against your site right now and covers AI crawlers — GPTBot, ClaudeBot, PerplexityBot — in addition to traditional search bots. It also checks AI-specific files (llms.txt, per-page markdown) and signals that Google Search Console doesn't surface.
Still have questions? Read our in-depth blog guides or contact us.
Run a free crawl test →