Robots.txt Validator

What the Robots.txt Validator checks

robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may request. It is a set of polite instructions: well-behaved bots follow it, but it is not access control, and it does not by itself remove pages from search results.

This validator fetches /robots.txt from the domain you enter and checks that it returns successfully. It looks for rules that block the entire site, rules that block CSS or JavaScript files search engines need to render your pages, and a Sitemap: directive. It also lists the known AI crawlers and shows whether your rules allow each one, grouped by role: search crawlers, user-triggered fetchers that read a page when someone asks an assistant about it, and bots that collect training data.

Why it matters

A single stray line, often Disallow: / left over from a staging server, can stop search engines from crawling a whole site. Blocking /assets/ or /wp-includes/ can prevent Google from rendering pages correctly, which affects how it understands layout and mobile usability.

AI crawlers add a newer decision. Many site owners want to appear in AI search answers while opting out of model training, and those are controlled by different user agents. Without checking, it is easy to block both or neither by accident. See our guide to AI crawlers.

How to fix common problems

Keep the file short and explicit. A sensible baseline is User-agent: * with Disallow: lines only for private areas such as admin or cart paths, plus Sitemap: https://example.com/sitemap.xml. Never block the CSS, JavaScript or image folders your pages use.

To keep a page out of search results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header instead. A page blocked in robots.txt can still be indexed from links, because the crawler never sees the noindex.

For AI, decide per role. To opt out of training while staying visible in AI search, add separate groups, for example User-agent: GPTBot with Disallow: / and User-agent: Google-Extended with Disallow: /, and leave search agents such as OAI-SearchBot allowed. Our robots.txt guide covers the syntax in detail. The full PerfoMetrix audit checks robots.txt alongside 170+ other checks, including your sitemap and page-level indexing tags.

Questions

Does Disallow remove a page from Google?

No. Disallow stops crawling, not indexing. Google can still list a blocked URL if other sites link to it. Use a noindex tag on a crawlable page to remove it.

What happens if robots.txt returns a 404?

Crawlers treat a missing file as permission to crawl everything. Server errors are different: a robots.txt that returns 5xx can make Google pause crawling the site.

Do all AI bots respect robots.txt?

The major vendors document user agents and say they honor robots.txt, but compliance is voluntary. For content that must stay private, use authentication rather than robots rules.

Can I stop this checker from scanning my site?

Yes. A group for User-agent: PerfoMetrixBot with Disallow: / makes our scanner refuse to scan your site.