Module 3 · 3.3

robots.txt: what you do and don't block

Module 3: The technical basics of a page · Reading time: 13 minutes

Learning objective: After this lesson you can read and write a robots.txt file, work out for yourself whether Googlebot may crawl an address, explain what happens when the file is missing or unreachable, and you know why robots.txt is neither security nor a noindex.

Key concepts: robots.txt (RFC 9309), crawler, group, User-agent, Disallow, Allow, longest match, equal length (Allow wins), * and $, Sitemap:, robots.txt status code (4xx, 5xx), 500 KiB, 24-hour cache, not security, not a noindex, AI crawlers (Google-Extended, GPTBot)

A sign by the front door

A robots.txt is a plain text file in the root folder of your site, always at the same address: https://www.example.com/robots.txt. A well-behaved crawler (lesson 1.2) requests that file first and reads which parts of the site it may not visit. It is a sign by the front door, not a lock: Googlebot and Bingbot respect it, but a malicious robot simply reads the sign and walks on in.

Since 2022 the protocol has been an official internet standard: RFC 9309. Google also describes exactly how Googlebot reads the file. We use both sources in this lesson.

A few fixed agreements:

The file sits in the root folder of the host. A robots.txt in /blog/robots.txt is not read.
It only applies to that host, that protocol and that port. https://www.example.com/robots.txt does not apply to https://shop.example.com/; that needs its own file.
It is plain text in UTF-8.

How the file is built

User-agent: *
Disallow: /winkelmand/
Disallow: /zoeken
Allow: /zoeken/tips

User-agent: Googlebot
Disallow: /concepten/

Sitemap: https://www.example.com/sitemap.xml

The file consists of groups. A group starts with one or more User-agent: lines, followed by the Disallow: (do not visit) and Allow: (may visit) rules. The path after Disallow: is the start of an address: Disallow: /zoeken (Dutch for "search") blocks /zoeken, /zoeken?q=rozen and also /zoekenmachine, because they all start with /zoeken.

The Sitemap: line does not belong to any group and tells crawlers where your sitemap is (lesson 3.4). Google requires a full address with https:// and the host name.

A Disallow: without a path blocks nothing. Disallow: / blocks everything.

Which group applies?

A crawler chooses one group: the one with the most specific name that matches it. So Googlebot follows the User-agent: Googlebot group if there is one, and then ignores the * group completely. In the example above, Googlebot may therefore crawl /winkelmand/ (the shopping basket): that rule is only in the * group. This is one of the most common mistakes: if you create a separate Googlebot group, you must repeat there every rule that should also apply to Google.

The name is not case-sensitive (googlebot = Googlebot). If there are two groups for the same crawler, Google merges them.

Which rule wins?

Within the group, an address can match several rules. Then:

The longest matching rule wins (the most specific path).
If an Allow and a Disallow are exactly the same length, Allow wins. Google calls this "the least restrictive rule".

Two characters have a special meaning:

Character	Meaning	Example	Matches	Does not match
`*`	zero or more of any characters	`Disallow: /*?sessie=`	`/rozen?sessie=12`	`/rozen?kleur=rood`
`$`	end of the address	`Disallow: /*.pdf$`	`/folder.pdf`	`/folder.pdf?v=2`

A worked example with the * group above, for the Bingbot crawler (which has no group of its own):

Address	Matching rules	Result
`/winkelmand/`	`Disallow: /winkelmand/`	not allowed
`/zoeken/tips`	`Disallow: /zoeken` (7 characters) and `Allow: /zoeken/tips` (12)	allowed: the longest wins
`/zoeken?q=rozen`	`Disallow: /zoeken`	not allowed
`/bruidsboeketten`	none	allowed
`/robots.txt`	(always)	allowed: robots.txt itself is always permitted

If the file is missing, or the server falters

What Google does depends on the status code your server returns for /robots.txt:

Response	What Google does
200 (found)	Reads the rules
Redirect (301, 302)	Follows at least five hops; after that Google treats it as a 404
4xx, such as 404 or 403 (except 429)	As if there is no robots.txt: everything is allowed
5xx or no connection	For the first 12 hours Google does not crawl the site and tries again; after that it uses the last good version from the cache for up to 30 days. If it still fails after that while the rest of the site works, Google acts as if there is no robots.txt

So a missing robots.txt is not a problem: everything is allowed. A robots.txt that returns a server error definitely is a problem: Google temporarily stops crawling. RFC 9309 is even stricter: if the file is unreachable, a crawler must assume that everything is disallowed.

Also:

Google reads at most 500 KiB; anything after that is ignored.
Google usually caches robots.txt for up to 24 hours, so a change does not take effect immediately. In Search Console, under Settings, you will find the robots.txt report: which version Google last fetched, with errors and warnings, and a button to request a recrawl.
Google only recognises User-agent, Allow, Disallow and Sitemap. Googlebot ignores rules such as Crawl-delay or Noindex.

What robots.txt does not do

It is not security

Anyone can read your robots.txt, including people with bad intentions. A rule such as Disallow: /beheer-oud/ actually points out where something interesting is. RFC 9309 says so itself: the rules are not a form of access control. To truly shield something, use a password or have the server refuse access, for example with .htaccess (see Further reading).

It is not a noindex

This is the most important point of the lesson. robots.txt controls crawling, not indexing (lesson 1.3). Google states plainly that robots.txt is not a way to keep a page out of Google. If other sites link to a blocked address, that address can still appear in the search results, just without a description.

If you want to keep a page out of the search results, use noindex (in a <meta name="robots"> tag or the X-Robots-Tag header) or put the page behind a password. And watch out for the classic trap: if you also Disallow a noindex page in robots.txt, Google cannot visit the page and so can never read the noindex.

Don't block parts Google needs

Don't block folders containing stylesheets, scripts or images that your page needs to display properly. Google views your page the way a browser does; without those files, Google understands the page less well.

AI crawlers

More and more companies collect web pages for artificial intelligence. You can address some of them in robots.txt:

Google-Extended is not a separate robot but a name in robots.txt that you use to indicate whether Google may use your content for training Gemini models and for grounding. Google states that this has no effect on your inclusion or position in Google Search.
GPTBot from OpenAI collects pages that may be used for training models. OAI-SearchBot is for ChatGPT's search feature: if you block it, your site will not appear in ChatGPT's search answers.

User-agent: GPTBot
Disallow: /

Whether you want that is a choice, not a technical necessity. If you accidentally block a search robot (Googlebot, Bingbot, OAI-SearchBot), you will be found less well there.

In practice: where is your robots.txt?

WebOké's AI Builder creates a robots.txt itself when you publish, with User-agent: *, Allow: / and a Sitemap: line. If you set the site to not be indexed, or it is a free demo, it contains Disallow: /.
WordPress creates a virtual robots.txt with a reference to the sitemap. If you put your own robots.txt file in the web folder, that takes precedence.
Your own site on a hosting package: put robots.txt in the web folder (in DirectAdmin, public_html) using File Manager or FTP.

Other providers work in a similar way: the file must always be reachable at /robots.txt on your host.

Summary

robots.txt lives at /robots.txt on each host and tells well-behaved crawlers what they may not visit.
A crawler follows one group (the most specific name, otherwise *). Within the group the longest rule wins; if they are the same length, Allow wins.
A 404 means "everything is allowed"; a server error makes Google temporarily stop crawling. Google reads 500 KiB and caches for up to 24 hours.
robots.txt is neither security nor a noindex. Don't block a noindex page in robots.txt.
You can handle AI crawlers such as GPTBot and the name Google-Extended separately, without consequences for Google Search.

Quiz

Exercise

Exercise A3.3: Your robots.txt checked. Enter your domain, plus three addresses on your own site (for example the homepage, a product page and your shopping basket or search page), and for each address your expectation: may Googlebot crawl it, yes or no? The program fetches your robots.txt, checks the status code, the size, the groups and the Sitemap: line, and works out for each address what Googlebot may do according to the rules in this lesson. Was your expectation right? No site of your own? Use www.wikipedia.org.

Cookies at WebOké

Cookie settings

Necessary

Marketing