Run an AI website audit for search readiness.

What does an AI website audit check?

AI Crawl Score is a free public website audit tool that checks crawl and content signals visible in HTML. Enter a URL and it requests robots.txt and any declared sitemap, then samples up to five public pages for response status, robots directives, title and description, canonical URL, headings, extracted text, JSON-LD syntax and basic form-control names. The report shows each observation and recommendation, including its source, weight and evidence limits. Use it to find technical issues and decide what to inspect next. The check does not access private analytics or server logs, prove a named crawler visited your site, render every page, or predict indexing, rankings, traffic or AI citations. Search and answer systems choose sources independently. Validate important results in first-party search tools and check performance with real-user or lab measurements. The score is a transparent technical diagnostic, not an outcome guarantee.

Understand robots.txt and AI crawler access

Your robots.txt file communicates crawling preferences, including different choices for search discovery and training collection. The AI crawler checker compares documented crawler tokens and shows the matched policy for the tested URL. The robots.txt AI checker helps you inspect groups, path rules and sitemap declarations. An allowed rule is evidence of a stated preference, not proof that a crawler visited your site. Server authentication, network conditions and CDN rules can still affect access. Preserve restrictions that protect content you deliberately exclude.

Review the content available without JavaScript

The scanner retrieves public HTML using its own identified client. It checks titles, descriptions, headings, canonical links and indexing directives in that response. This gives you a starting point for understanding content extractability. A page that depends on JavaScript may require a separate rendered browser review. The agent readiness checker also inspects native controls and apparent accessible names. These markup observations help you identify follow-up work, but they cannot confirm keyboard behavior, computed visibility or how an interactive application behaves.

Inspect sitemaps and structured data with clear limits

The sitemap health checker discovers a sitemap and inspects XML within a bounded budget. It can help you review structure and host consistency while distinguishing parsed URLs from pages actually tested. The structured data checker examines JSON-LD syntax and declared types in retrieved HTML. Valid JSON alone does not establish complete schema compliance or eligibility for a search feature. Review the specific evidence before making changes, and use the relevant provider documentation when your page needs a more specialized validation.

Choose focused checks and optional generators

Use dedicated checks for ChatGPT search readiness, OAI-SearchBot, GPTBot, Claude and Perplexity when you need to investigate one crawler policy. These tools share the same analysis engine and explain their different purposes. The AI robots.txt policy generator creates a snippet to review and merge with your existing rules. The llms.txt generator creates an optional documentation index. Neither generator publishes changes for you. Missing llms.txt or blocking training collection alone does not reduce your search readiness score.

Turn findings into a repeatable review

A free scan samples up to five public pages, so its score describes that sample rather than an entire website. Read failed, unknown and untested findings separately. Download the JSON report to keep a local record, make a deliberate change and run another scan when capacity permits. Submitted targets and fetched page bodies are excluded from application logs. Anonymous reports are not saved or shared automatically. Public documentation explains the scoring methodology, data sources and safe scanning boundaries so your team can assess the result.

Each scan requests public resources again; it does not promise continuous monitoring or automatic updates between visits. Crawler source reviews and availability checks support maintenance, while factual changes require review. Accounts, ownership verification, history and alerts remain planned capabilities. The public tools remain available individually, with clear explanations of their scope and limitations. The useful outcome is a clearer technical checklist, with evidence you can inspect and limits you can understand, not a guarantee of rankings, indexing, traffic or citations from any provider.