Robots.txt Generator

What the Robots.txt Generator does

robots.txt is a plain text file at the root of your domain, such as https://example.com/robots.txt, that tells crawlers which paths they may request. It is made of groups: one or more User-agent lines naming the crawlers, followed by Allow and Disallow rules for paths, plus optional Sitemap lines with absolute URLs.

This generator builds the file from your choices: paths to block or allow for all crawlers, the sitemap URLs to list, and an option to opt out of AI training crawlers while staying visible in search and AI search. With that option on, it adds a separate group with Disallow: / for training agents such as GPTBot, ClaudeBot and CCBot and for the Google-Extended token, and leaves search crawlers alone.

Why it matters

A correct robots.txt keeps crawlers out of areas that waste their time, such as internal search results, cart and admin paths, and points them to your sitemap. A wrong one can do real damage: a leftover Disallow: / hides an entire site from search, and blocking CSS or JavaScript folders stops search engines from rendering pages properly.

AI adds a decision robots.txt was never designed for. Training crawlers and AI search crawlers use different user agents, so you can choose separately whether your content may train models and whether it may be cited in answers. Writing those groups by hand makes it easy to block both or neither. See our guides to robots.txt and AI crawlers.

How to fix common problems

Save the output as a file named robots.txt and upload it to the root of the host, so it loads at /robots.txt with status 200 as plain text. Each host needs its own file: example.com, www.example.com and shop.example.com are read separately. In WordPress, a physical file in the web root replaces the virtual one WordPress generates, and some SEO plugins offer an editor for it. List the same sitemap URLs you submit in Search Console (XML sitemaps).

Do not use robots.txt to hide private pages. The file is public and names the paths you list, and a blocked URL can still be indexed from links. Protect private areas with authentication, and use a noindex tag on crawlable pages you want kept out of results.

After uploading, run the Robots.txt Validator to confirm the file is reachable and does not block your whole site or its CSS and JavaScript, and the AI Crawler Checker to see each AI agent’s access. The full PerfoMetrix audit rechecks robots.txt every time you scan.

Questions

Will blocking AI training crawlers hurt my Google rankings?

No. Google-Extended only controls whether content may be used for Gemini and does not affect Google Search. Do not block Googlebot as an AI opt-out: it is the crawler for Google Search itself.

If Allow and Disallow both match, which wins?

Under the robots.txt standard (RFC 9309), which Google follows, the most specific rule wins, meaning the one with the longest matching path. When an Allow and a Disallow rule are equally specific, Allow wins.

Does robots.txt support Crawl-delay?

Google ignores Crawl-delay. Some other crawlers, such as Bingbot, read it.

How fast do changes take effect?

Google generally caches robots.txt for up to 24 hours, and other crawlers have their own schedules, so allow a day before judging the effect of a change.