robots.txt vs noindex: How to Keep a Page Out of Google

You have a page that shouldn't show up in Google: a thank-you page, internal search results, an old landing page, a staging copy of your site. The instinct is to block it in robots.txt. Very often that is exactly the wrong move, and it can keep the page in search results longer.

The reason is simple once you see it. robots.txt and noindex control two different steps: crawling and indexing.

Crawling vs. indexing

• Crawling is Googlebot fetching a URL to see what's there.
• Indexing is Google storing that page so it can appear in search results.

robots.txt controls crawling. A Disallow rule tells well-behaved crawlers not to fetch certain URLs. It says nothing about whether those URLs can appear in results.

noindex controls indexing. It tells search engines that support it not to show the page in results. But it lives inside the page (or its HTTP headers), so a crawler has to fetch the page to see it.

Google's own robots.txt introduction puts it plainly: robots.txt "is not a mechanism for keeping a web page out of Google."

Why blocking a page in robots.txt can keep it in Google

Suppose you add this to robots.txt:

User-agent: *
Disallow: /old-pricing

Googlebot stops fetching /old-pricing. But if other pages, on your site or elsewhere, still link to it, Google knows the URL exists. It can index the URL without ever reading the page, and it shows up in results without a description, because Google never saw the content.

Now suppose you also add a noindex tag to that page. It doesn't help: Googlebot is blocked from fetching the page, so it never sees the tag. Google's noindex documentation is explicit that for noindex to work, the page "must not be blocked by a robots.txt file."

In Search Console, this situation often shows up in the Page indexing report as "Indexed, though blocked by robots.txt."

How to keep a page out of Google with noindex

There are two ways to send a noindex rule. Use whichever fits the content.

The robots meta tag (HTML pages)

Put this in the page's <head>:

<meta name="robots" content="noindex">

To target only Google, use name="googlebot" instead of name="robots". You can build the tag along with the rest of your head tags in our Meta Tag Generator.

The X-Robots-Tag header (any file type)

For PDFs, images and other files that have no HTML head, send the rule as an HTTP response header:

X-Robots-Tag: noindex

In nginx, for example, you could apply it to every PDF:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}

Headers are invisible in the browser, so check that it's actually being sent. Our HTTP Headers Viewer lists every response header for a URL, including any X-Robots-Tag.

Then wait, or ask for a recrawl

noindex takes effect when Google next crawls the page. Google notes that it "may take months for Googlebot to revisit a page." To speed things up, open the URL in Search Console's URL Inspection tool, confirm that Googlebot sees the noindex in the HTML it fetched, and request indexing so it gets recrawled.

What robots.txt is actually good for

robots.txt is still useful. It just answers a different question: which URLs should crawlers spend time on? Good uses include:

• Endless URL variations, such as sort, filter and session parameters in an online shop, that would otherwise eat up crawling
• Internal search result pages that can generate an unlimited number of URLs
• Cart, checkout and account areas that have no value in search
• Images, video and audio files you don't want in Google's image or video results, which Google says robots.txt can handle
• Pointing to your sitemap with a Sitemap: line

The format is standardized as the Robots Exclusion Protocol in RFC 9309. Our Robots.txt Generator follows it and includes a tester that tells you whether a given path is allowed or blocked for a given crawler, and which rule decided it. That's handy because the matching rules aren't always obvious: the longest matching rule wins, and Allow wins a tie.

One thing to never put in robots.txt: Noindex:. In Google's words, "specifying the noindex rule in the robots.txt file is not supported," so the line does nothing.

Which one to use: common situations

Fixing a page you already blocked

If you blocked a URL in robots.txt and it's still showing in Google, the fix feels backwards:

1. Remove the Disallow rule for that URL, so Googlebot can fetch it again.
2. Add a noindex meta tag or header, or make the URL return 404 or 410 if the page is gone for good.
3. Request a recrawl with URL Inspection in Search Console, and confirm the noindex or 404 is what Googlebot sees.
4. Wait until the URL drops out of the index. If you still want to save crawl time, you can add the Disallow rule back after that.

For anything urgent, Google's Removals tool hides a URL quickly, but "a successful request lasts only about six months." It's a stopgap while the permanent fix takes effect.

Mistakes that keep pages in search

• Disallow plus noindex on the same URL. The Disallow hides the noindex. Pick noindex.
• Leaving a "block everything" robots.txt after launch. A Disallow: / from a staging setup that sneaks into production stops crawling of the whole site.
• Putting secrets in robots.txt. Anyone can read /robots.txt, so listing /secret-admin/ there advertises it.
• Noindex in the sitemap. A sitemap tells Google which URLs you want indexed. Listing noindex pages sends mixed signals, so leave them out.
• Expecting every search engine to behave like Google. Google notes that some search engines might interpret noindex differently, and not all crawlers obey robots.txt.

FAQ

Does noindex stop Google from crawling the page?

No. Google still needs to crawl the page to see the noindex rule, and it may keep crawling it occasionally, though usually less often over time.

Should I use noindex, follow or just noindex?

noindex on its own is enough. According to Google's robots meta tag documentation, the default is all, meaning no restrictions on indexing or following links, so adding follow changes nothing.

Can I block a page in robots.txt and still have it indexed?

Yes, and that's the problem described above. If other pages link to it, Google can index the URL without its content, which usually gives a poor-looking result.

How do I check what Google sees?

Use URL Inspection in Search Console. It shows whether the URL is blocked by robots.txt, whether a noindex was detected, and the HTML Googlebot received.

• #Crawling
• #Google Search Console
• #noindex
• #robots.txt
• #Technical SEO