PerfoMetrix

All articles · Technical SEO

robots.txt Mistakes That Quietly Hurt Your Search Visibility

By PerfoMetrix Team5 min read

Cover with the article title next to a gate across a path with a small robot in front of it, in violet on a dark background

robots.txt is one of the oldest files on the web and one of the easiest to get wrong. It’s a plain text file with a handful of directives, so nobody reviews it. Then a deploy copies the wrong version, or someone adds a “harmless” rule, and crawling of half the site stops without a single error in your logs.

The damage is quiet because robots.txt failures don’t break anything for users. Pages load, forms work, monitoring stays green. The first sign is usually a dip in Search Console weeks later. This article goes through the mistakes we see most often, why each one hurts, and how to check for it.

What robots.txt actually controls

robots.txt tells well-behaved crawlers which URLs they may fetch. That’s all. It was standardized in 2022 as RFC 9309, after decades as an informal convention, and the core rules are simple:

  • It lives at /robots.txt on each host, and applies only to that host, protocol and port. https://example.com/robots.txt says nothing about https://www.example.com or https://shop.example.com.
  • It’s organized in groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules.
  • Paths are matched as prefixes, are case-sensitive, and support * (any sequence) and $ (end of URL).

What it doesn’t control is whether a URL appears in search results. That distinction is behind the most damaging mistake on this list.

You don’t want /internal-search or /thank-you in Google, so you disallow them. Months later they still show up as bare URLs with “No information is available for this page”.

That’s expected. Disallow stops Google from crawling the page, not from indexing the URL. If other pages link to it, Google can index the address without ever seeing the content. Worse, because the page can’t be fetched, Google can’t see a noindex tag on it either.

The fix is the opposite of what feels natural:

  1. Remove the Disallow rule so the page can be crawled.
  2. Add <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header.
  3. Once the URLs have dropped out, you can decide whether to block crawling again. Usually there’s no need.

Our noindex guide covers the header and meta variants. Also note that a Noindex: line inside robots.txt does nothing in Google. It was never part of the standard, and Google stopped honoring it in 2019.

Mistake 2: Shipping the staging file to production

Staging environments often carry this, quite rightly:

User-agent: *
Disallow: /

If your build copies a static robots.txt from the repository, sooner or later that version reaches production. Everything is blocked, and nothing visibly breaks.

Two habits prevent it:

  • Generate robots.txt per environment instead of committing one file. The production host gets the real rules; every other host gets Disallow: /, ideally backed by an X-Robots-Tag: noindex header so a leaked staging URL can’t be indexed either.
  • Test it after every deploy. A check that fetches the production robots.txt and fails if it contains Disallow: / for * or Googlebot costs a few lines in CI.

Mistake 3: Blocking CSS, JavaScript and images

Older advice said to keep crawlers out of /assets/, /wp-includes/ or /static/ to “save crawl budget”. Google renders pages much like a browser does. If it can’t fetch your stylesheets and scripts, it may see an unstyled page, miss content that JavaScript inserts, or fail to judge whether the page works on mobile.

Let crawlers fetch everything a page needs to render. If you block anything in those directories, make sure it’s genuinely unused by public pages, such as an admin bundle.

Mistake 4: Misunderstanding how groups combine

This file looks like it blocks /private/ for everyone and adds a tweak for Googlebot:

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /tmp/

It doesn’t. A crawler follows only the most specific group that matches its name. Googlebot finds its own group and ignores the * group entirely, so for Googlebot /private/ is open. If a crawler has its own group, repeat every rule it should follow there:

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /private/
Disallow: /tmp/

This matters more now that many sites add groups for AI crawlers. Each new group is a full rule set, not an addition to the default one.

Mistake 5: Rules that match more, or less, than you think

Prefix matching catches people out in both directions.

  • Disallow: /blog blocks /blog, /blog/ and every post, but also /blog-archive and /blogroll.html. Use Disallow: /blog/ if you only mean the directory.
  • Disallow: /*.pdf blocks /report.pdf but also /report.pdf?download=1 and /a.pdf-viewer/. Add $ to anchor the end: Disallow: /*.pdf$.
  • Disallow: /Admin/ doesn’t block /admin/. Paths are case-sensitive.
  • An empty Disallow: allows everything. It’s not a typo for Disallow: /, though it’s often read as one.

When Allow and Disallow both match, Google applies the rule with the longest matching path, and on a tie the less restrictive one (Allow) wins:

User-agent: *
Disallow: /account/
Allow: /account/signup

Here /account/signup stays crawlable while the rest of /account/ is blocked.

Mistake 6: Ignoring status codes and unsupported directives

Status codes change the meaning

What the server returns for /robots.txt matters as much as what’s in it:

  • 200: the rules are applied.
  • 404 or another 4xx: crawlers act as if there were no restrictions and crawl everything. Fine if that’s what you intend, surprising if a rewrite rule broke the file.
  • 5xx: crawlers treat the whole site as temporarily disallowed. A robots.txt route that times out or errors under load can pause crawling of every page, even though the pages themselves are fine.
  • Redirects are followed for a limited number of hops (RFC 9309 asks crawlers to follow at least five). A redirect to an HTML login page or a CDN error page is parsed as robots.txt and produces nonsense.

Serve robots.txt as a static, cached text/plain response that doesn’t depend on your database or session layer. Google’s robots.txt documentation also notes a 500 KiB size limit: anything after that is ignored, which only bites generated files with thousands of lines.

Directives Google doesn’t support

A few lines show up in robots.txt files everywhere and do nothing in Google:

  • Crawl-delay is ignored by Googlebot. Some other crawlers honor it. To slow Googlebot down, return 503 or 429 temporarily when you’re overloaded.
  • Noindex, as above.
  • Comments that explain intent are fine (#), but they’re not instructions.

The Sitemap: line, on the other hand, is supported and useful. It takes an absolute URL and can appear anywhere in the file, outside any group:

Sitemap: https://example.com/sitemap.xml

See our XML sitemap guide for what that file should contain.

Checking your own file

A short review covers most of the risk:

  1. Fetch https://<each host>/robots.txt directly and confirm a 200 with text/plain.
  2. Read every group as the crawler would: find the most specific matching User-agent, ignore the rest, and walk the rules for your key URL patterns.
  3. Make sure nothing needed for rendering is blocked.
  4. Look for a Disallow on anything you actually want out of the index, and replace it with noindex.
  5. Use the robots.txt report in Search Console to see which version Google last fetched and whether it had errors.
  6. Add a deploy check so the staging file can never reach production unnoticed.

Our robots.txt guide summarizes the syntax. The file is small, but it sits in front of every crawl of your site, so it deserves the same review as any other production config.

Tags:robots.txtcrawlingindexingGooglebottechnical SEO