Keeping a page out of Google: robots.txt or noindex?

Don't want a page to show up in Google, for example a thank you page, a test page, or an old promotion? A lot of people then think of the file robots.txt. But robots.txt doesn't keep a page out of Google. That's what noindex is for. In this article, you'll read the difference, how to use both, and what to do with a page you've deleted. We'll use example.com as an example.

The difference in short

Neither of these is a form of security. Anyone can read your robots.txt, and a page with noindex remains perfectly reachable. If something really needs to be shielded off, put it behind a password. See Securing a folder with a password in DirectAdmin.

Using noindex

Put this line in the <head> of the page:

<meta name="robots" content="noindex">

For files without HTML, such as a pdf, the server can send the same instruction as a header: X-Robots-Tag: noindex. The effect is the same.

Usually you don't have to type this yourself:

• AI Builder: open the SEO section, choose the page and under More for this page turn off the Make this page findable in Google switch. The page remains reachable, but gets a noindex. Then publish again.
• WordPress: an SEO plugin has a setting per page to prevent the page from being indexed. The checkbox Discourage search engines from indexing this site under Settings, Reading applies to the whole site.
• Other website builders: look for a setting like "hide from search engines".

Creating or editing robots.txt

On a hosting package, robots.txt is located in the web folder of your domain, in DirectAdmin the folder public_html. You create or edit the file via File Manager or FTP, see Opening File Manager: the folders in your package. An example:

User-agent: *
Disallow: /zoeken
Disallow: /winkelmand/

Sitemap: https://example.com/sitemap.xml

• User-agent: * applies to all search robots. Disallow: with a path forbids anything starting with that path. Disallow: / forbids the entire site.
• If you also create a group User-agent: Googlebot, Google only follows that group and ignores the * group. So repeat there all the rules that also apply to Google.
• Don't block folders with stylesheets, scripts, or images that your page needs.
• No robots.txt is not a problem: then everything is allowed. If the address returns a server error, Google temporarily stops fetching.

WordPress creates its own robots.txt as long as there's no file in the web folder. The AI Builder sets its own robots.txt with a Sitemap: line when publishing.

Google usually caches your robots.txt for up to 24 hours. In Search Console, under Settings, you'll find the robots.txt report: which version Google last fetched, with errors and warnings, and a button to have it fetched again.

A deleted page: 404 or 410, not a soft 404

If you've deleted a page, the address should give the status code 404 (not found) or 410 (gone). On the next visit, Google will then remove the address from the search results. If the page has a successor, redirect the old address with a 301 to that new page, see Site Redirect: redirecting a domain or page.

A soft 404 is a page that says it isn't there, or is nearly empty, but still gives the code 200 (found). Google often recognizes this and won't include such a page. In Search Console, you'll see it as the reason Soft 404. Redirecting all deleted pages to your homepage can also count as a soft 404. So make sure a page that's really gone gives a real 404 or 410.

Does a page need to come out of the results quickly, for example because it accidentally contained personal data? Then use Removals in Search Console. That works temporarily, for about six months. So also make sure the page has a 404, 410, password, or noindex.

Further reading

• Google: Block indexing with noindex (English)
• Google: Introduction to robots.txt (English)
• Google: robots.txt report
• Blocking files via .htaccess

Using WordPress? In the lesson Findability: the basics of the free WordPress course, you'll read more about robots.txt and the Discourage search engines checkbox.