robots.txt: what it does and how to set it up
robots.txt is a plain text file at the root of a host, such as https://example.com/robots.txt.
It tells crawlers which paths they may fetch and where to find your sitemap. The format is
standardized in RFC 9309.
User-agent: *
Disallow: /cart/
Disallow: /search
Sitemap: https://example.com/sitemap.xml
Why it matters
A working robots.txt lets you keep crawlers away from URLs that waste their time, such as internal
search results, filters that create endless URL combinations, or cart and checkout steps. Its
Sitemap: line is also one of the simplest ways for search engines to find your sitemap.
How a missing or broken file is handled matters too. Google treats a 404 or other 4xx response (except 429) as “no restrictions” and crawls everything. A server error (5xx) is different: Google treats it as a temporary signal to stop crawling the site until the file can be fetched again. A robots.txt that times out or returns 500 can therefore slow down or pause crawling of your whole site.
What we check
We request /robots.txt on the host of the scanned page (following up to two redirects).
| Result | When |
|---|---|
| Pass | the file returns 200 OK |
| Needs attention | it returns any other status, such as 404 or 500 |
| Couldn’t test | the request failed (for example, a timeout) |
If the file lists Sitemap: lines, we use them for the XML sitemap check.
We do not evaluate your Allow and Disallow rules for search engines.
How to fix
Create a file named robots.txt at the web root, served as text/plain with status 200. If you do
not want to block anything, this is a complete, valid file:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
An empty Disallow: means “nothing is disallowed”. Each host needs its own file: www.example.com
and shop.example.com do not share one.
Static file. Place it in the folder your server serves as /, for example public/ in most
frameworks.
Nginx, if you prefer not to keep a file on disk:
location = /robots.txt {
default_type text/plain;
return 200 "User-agent: *\nDisallow:\n\nSitemap: https://example.com/sitemap.xml\n";
}
WordPress generates a virtual robots.txt by default; SEO plugins let you edit it. A physical
robots.txt file in the web root takes precedence.
If the file returns 5xx or times out, fix that first: it can affect crawling of the entire site.
Rules worth knowing
- Disallow is not noindex. A disallowed URL can still appear in search results, usually without
a description, if other sites link to it. To keep a page out of search, allow crawling and use
noindexinstead. See the noindex guide. - Do not block CSS and JavaScript that pages need to render. Search engines render pages and need those files to see the layout and content.
- robots.txt is public. Anyone can read it, so do not list secret paths in it. Protect private areas with authentication.
- Rules are prefix matches and case-sensitive.
Disallow: /searchalso blocks/search-tips. - Size limit. Google reads only the first 500 KiB of the file.
- Not all bots obey it. Well-behaved crawlers follow it; abusive ones ignore it.
Opting out of PerfoMetrixBot
Our scanner checks a single page when a person asks for a report. It does not crawl your site. If you do not want your site scanned at all, add a group that names our bot explicitly:
User-agent: PerfoMetrixBot
Disallow: /
We read /robots.txt on the submitted URL’s host before requesting the page itself, and refuse the
scan if this rule is present. The group must name PerfoMetrixBot; a User-agent: * group does not
opt out, because a single on-demand check is not crawling.
Google’s introduction to robots.txt covers syntax and testing in more detail.