All articles · Site Operations

AI Crawlers and Your Site: Search, User Fetchers and Training Bots

By PerfoMetrix Team·October 30, 2026·6 min read

A few years ago, the bots in your access logs were mostly search engines, SEO tools and the odd scraper. Now there’s a growing list of user agents from AI companies, and a common reaction is to block all of them, or none of them, without looking closer.

Both choices can cost you something. AI crawlers don’t all do the same job. Some build a search index that sends visitors back to you, some fetch a page because a person asked an assistant about it, and some collect text to train models. Treating them as one group means either giving away content you’d rather keep or disappearing from tools your audience already uses. This article sorts them by role and shows how to set a policy for each.

Three roles, not one

The useful way to think about AI crawlers is by what happens with the content they fetch.

1. Search crawlers build an index for an AI search or answer product. When the product answers a question using your page, it can cite and link to you. Blocking them is roughly like blocking a search engine.
2. User-initiated fetchers load a specific page because a person asked for it right then, for example by pasting your URL into a chat or asking the assistant to read a page. These behave more like a browser acting on someone’s behalf than like a crawler.
3. Training crawlers collect content to train or improve models. Their output doesn’t link back to any particular page, and blocking them generally has no effect on whether you appear in search or answer products.

Several companies now run separate user agents for these roles, which is what makes a nuanced robots.txt possible at all.

The user agents worth knowing

These are the well-documented ones. Each operator publishes its own documentation, and the details change, so treat this as a map rather than a final reference.

CCBot deserves a note. Common Crawl is a non-profit that publishes a free archive of the web. It isn’t an AI company, but its dataset is one of the most common sources of training text, so sites that want to opt out of training usually include it.

Google-Extended is a switch, not a bot

Google doesn’t send a separate crawler for Gemini. Regular Googlebot fetches your pages, and the Google-Extended token in robots.txt controls whether that content may be used to train and ground Google’s Gemini models. Google lists it among its common crawlers and fetchers.

Two consequences follow:

• Disallowing Google-Extended does not remove you from Google Search, and Google states it doesn’t affect ranking.
• It also doesn’t control AI features inside Google Search, such as AI Overviews. Those are part of Search, and the controls for them are the normal Search ones, like nosnippet and Googlebot’s access.

If you want to stay in Google Search but keep your content out of Gemini training, disallowing Google-Extended is the tool for that. If you want to stay out of AI Overviews specifically, there’s no separate token; it’s a trade-off with your regular search snippets.

Writing the policy in robots.txt

Here’s a common middle-ground policy: stay visible in AI search and answer products, allow fetches that a person asked for, and opt out of training.

# Default for everyone else
User-agent: *
Disallow: /account/
Disallow: /cart/

# AI search and user-initiated fetches: same rules as everyone
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: PerplexityBot
Disallow: /account/
Disallow: /cart/

# Training: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

A few details make or break a file like this:

• A crawler follows only its most specific group. Once OAI-SearchBot has its own group, it ignores the * rules entirely, which is why the shared disallows are repeated. A bare User-agent: GPTBot group with only Disallow: /private/ would leave everything else open to it.
• Several User-agent lines can share one group, as above. That keeps the file short and makes the policy readable by role.
• robots.txt applies per host. If your content also lives on docs. or blog. subdomains, each needs its own file.

Our robots.txt guide covers the matching rules in more detail, and the AI crawlers guide summarizes what each role means for visibility.

What robots.txt can’t do

robots.txt is a voluntary protocol, standardized in RFC 9309. The major operators listed above say they honor it for their crawlers, but the file doesn’t enforce anything. Keep a few limits in mind:

• It isn’t retroactive. Blocking a training crawler stops future fetches; it doesn’t remove anything already collected.
• User fetchers may be treated differently. Because they act on a specific person’s request, some operators document different robots.txt behavior for them than for their automated crawlers. Read each operator’s current documentation rather than assuming.
• User agent strings can be faked. Anyone can send GPTBot in a header. Several operators publish the IP ranges their crawlers use, so if you make decisions in your firewall or CDN, verify the IP, not just the name.
• Unknown bots ignore all of this. Rate limiting and bot management at your CDN handle crawlers that don’t identify themselves or don’t respect the file.

If you need content kept out of any AI system, robots.txt alone isn’t enough. Put it behind authentication.

Where llms.txt fits

llms.txt is a community proposal from 2024: a Markdown file at /llms.txt that gives language models a short summary of a site and links to its most useful pages, sometimes with plain-text versions. It’s simple to publish and can help tools that choose to read it, such as some coding assistants working with documentation sites.

It’s worth being clear about its status. It isn’t a standard, it doesn’t control access the way robots.txt does, and major search engines don’t use it as a ranking signal. If your site has documentation developers query through AI tools, an llms.txt is a cheap, low-risk addition. Otherwise, it’s optional, and your time is better spent on pages that are clearly structured and readable in plain HTML, which every crawler benefits from. Our guide on AI-readable content covers that side.

Deciding on your policy

A few questions settle most cases:

• Do you want to be cited in AI answers? If your audience uses ChatGPT search, Perplexity or similar tools to research products or topics like yours, allow the search crawlers and user fetchers. Blocking them removes you from those answers, and the citations are one of the few ways they send traffic back.
• Do you object to training use? That’s a business and licensing decision, not an SEO one. Opting out of training through GPTBot, ClaudeBot, CCBot and Google-Extended doesn’t remove you from search products.
• Is the content behind a paywall or licensed? Then the paywall, not robots.txt, is what protects it.

Whatever you decide, write it down next to the robots.txt file, check your logs every few months for new user agents, and confirm the file still says what you meant after every deploy. A scan with PerfoMetrix shows which of these roles your current robots.txt allows or blocks, which is a quick way to see whether the file matches the policy you intended.

Tags:AI crawlersrobots.txtGPTBotClaudeBotllms.txt