AI Crawlers and robots.txt: Search, Assistants and Training
AI assistants such as ChatGPT, Claude, Perplexity, Gemini and Copilot answer questions using web pages, and increasingly they cite and link to them. Whether your pages can be among those sources depends first on a simple question: are their crawlers allowed in?
AI companies run different crawlers for different jobs, and robots.txt lets you treat them differently. Blocking the wrong one by accident is common, often through a template copied from somewhere or a “block AI” setting turned on without reading the details.
Three kinds of crawler
| Role | What it does | Examples |
|---|---|---|
| Search | Builds the index that AI answers and search results draw from | Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User fetchers | Loads a specific page when a person asks an assistant about it | ChatGPT-User, Claude-User, Perplexity-User |
| Training | Collects pages to train AI models | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider |
Two details trip people up:
- Google AI Overviews and Bing Copilot use the normal search crawlers. There is no separate “AI search” bot to allow for them; blocking Googlebot or Bingbot removes you from both search and their AI answers.
- Google-Extended and Applebot-Extended aren’t separate crawlers. They are robots.txt tokens that control whether content Google and Apple already crawl may be used for their AI models. Blocking Google-Extended does not affect Google Search.
What we check
We read your robots.txt (the copy we already fetched for the scan) and decide, for each crawler,
whether it’s blocked from the whole site. We use the group that names the crawler, or the *
group if none does, and count a site-wide Disallow: / (or /*) that isn’t reopened with
Allow: /. Rules for individual folders aren’t counted as a block.
| Check | Passed | Needs attention | Failed |
|---|---|---|---|
| Search crawlers | none blocked | an AI search bot is blocked | Googlebot or Bingbot is blocked |
| Assistants can open your pages | none blocked | a user fetcher is blocked | |
| AI training crawlers | reported for information only: blocking them is your choice |
Members see a full table of every crawler and its access in the report.
Allowing search while opting out of training
A common, reasonable setup: appear in AI search and answers, but keep content out of model training.
# Training: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /
# Everyone else, including search crawlers and assistants
User-agent: *
Allow: /
Each crawler follows only the most specific group that names it, so the training bots obey the first group and everything else uses the second.
robots.txt isn’t the only switch
CDNs and firewalls can block bots before they ever see robots.txt. Cloudflare, for example, offers settings that block AI crawlers across a site. We can’t see those from outside, so if this check passes but AI tools still can’t read your pages, check your CDN’s bot settings and firewall rules as well.
Further reading: OpenAI crawlers, Google’s common crawlers and our robots.txt guide.