AI Crawler Checker

What the AI Crawler Checker checks

AI companies run several kinds of bots, and they do different jobs. Search crawlers such as OAI-SearchBot and PerplexityBot index pages so AI search features can cite them. User-initiated fetchers such as ChatGPT-User and Claude-User load a page when a person asks an assistant about it. Training crawlers such as GPTBot, ClaudeBot and CCBot (Common Crawl) collect content that may be used to train models. Google-Extended is a special case: it is not a crawler but a robots.txt control token that tells Google whether content crawled by Googlebot may be used for its Gemini models.

This checker reads /robots.txt on the domain you enter and works out, for each known AI user agent, whether your rules allow it, grouped by role. It flags an inconsistent policy, such as blocking one vendor’s training crawler while allowing another’s, and reports whether the site publishes an /llms.txt file.

Why it matters

The choice is no longer simply allowing or blocking AI. Many publishers want to appear in AI search answers, with a link back, while keeping their content out of training data. Those are controlled by different user agents, so a blanket rule often achieves the opposite of what was intended: blocking every bot can remove you from AI search, while a rule for GPTBot alone leaves other training crawlers free.

User-initiated fetchers deserve a separate thought. They act for a person who asked about your page, and some vendors state that robots.txt may not apply to them the way it applies to automated crawlers. robots.txt is a voluntary standard, so treat it as a statement of policy, not access control. See our guide to AI crawlers.

How to fix common problems

Decide per role, then write one group per user agent. To stay visible in AI search while opting out of training, add a group with Disallow: / for each training agent you want to exclude, such as User-agent: GPTBot, User-agent: ClaudeBot, User-agent: CCBot and User-agent: Google-Extended, and leave search agents such as OAI-SearchBot and PerplexityBot allowed. A crawler follows the most specific group that names it, so a named group replaces your User-agent: * rules for that bot rather than adding to them.

Blocking Google-Extended does not affect Google Search or your rankings; it only controls use in Gemini. Never block Googlebot as an AI opt-out: that removes you from Google Search itself, including its AI features. The syntax is covered in our robots.txt guide.

Re-check after every change and review the rules every few months, because vendors add and rename agents. The Robots.txt Generator can write the groups for you. An /llms.txt file is optional and informational; what matters more is that your content is in the HTML (AI-readable content). The full PerfoMetrix audit scores AI readiness, including content, schema and authorship.

Questions

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses OAI-SearchBot for search and GPTBot for training, and controls them separately. Block GPTBot to opt out of training and keep OAI-SearchBot allowed to stay eligible for ChatGPT search results.

Is Google-Extended a crawler?

No. It is a robots.txt token. Google crawls with its normal Googlebot and checks your Google-Extended rules to decide whether the content may be used for Gemini, so Google-Extended never appears in your server logs.

Do AI crawlers have to obey robots.txt?

robots.txt is voluntary. The major vendors document their user agents and say their crawlers honor it, but public content can still be fetched by bots that ignore it. Use authentication for content that must stay private.