Resources
AI Crawlability Blog
Guides on making your website visible to AI agents, crawlers, and AI-powered search engines.
Link Response Headers for Agent Discovery - RFC 8288 Explained
How HTTP Link headers (RFC 8288) let AI agents discover your site's key resources without parsing HTML - and how to add them to your site in minutes.
May 14, 2026 · 6 min readTechnicalMCP Server Cards: How AI Agents Discover Your Site's Tools
What the Model Context Protocol server card is, why /.well-known/mcp.json matters, and how to publish one so AI agents can discover and use your site's tools and data.
May 14, 2026 · 7 min readTechnicalAPI, OAuth, and OIDC Discovery for AI Agents: A Practical Guide
How AI agents discover your API capabilities and authentication requirements - covering OpenAPI specs, ai-plugin.json, OAuth authorization server metadata (RFC 8414), and OIDC discovery.
May 14, 2026 · 8 min readGuidesPer-Page Markdown Files: The Easiest Win for AI Crawlability
Learn how adding a .md companion file to each page gives AI agents clean, distraction-free content - and why this simple technique dramatically improves how AI systems read and cite your site.
May 6, 2025 · 7 min readGuidesThe Complete Guide to AI Crawlability in 2025
Everything you need to make your website visible to AI search engines and agents. A comprehensive checklist covering llms.txt, robots.txt, sitemaps, structured data, Open Graph, and per-page markdown.
April 29, 2025 · 12 min readTechnical SEOJSON-LD and Structured Data: Making Your Content AI-Readable
A practical guide to implementing JSON-LD structured data markup on your website to improve visibility in AI search engines, rich results, and knowledge graphs.
April 22, 2025 · 10 min readGuidesrobots.txt for AI Crawlers: A Complete 2025 Guide
A comprehensive guide to configuring robots.txt for AI bots in 2025. Learn which user agents to allow, how to protect sensitive paths, and best practices for AI-friendly crawler configuration.
April 15, 2025 · 9 min readSEOHow to Optimize Your Website for AI Search Engines
AI-powered search engines like Perplexity, ChatGPT Browse, and Claude work very differently from Google. Here's how to optimize your site for the new paradigm of answer engine optimization.
April 8, 2025 · 8 min readGuidesWhat is llms.txt and Why Every Website Needs One
Learn about the llms.txt specification - the emerging standard that helps AI agents and large language models understand and navigate your website's content.
April 1, 2025 · 7 min readCheck your site now
See how your site scores on AI crawlability - instantly, for free.