Module 1 · 1.3

Indexing: what gets into the index and what doesn't

Module 1: How search engines work · Reading time: 11 minutes

Learning objective: After this lesson you can judge whether a page can get into the index and, if not, which of the four "locks" is stopping it: the status code, robots.txt, noindex or the canonical.

Key concepts: indexing, Google's technical requirements, noindex (meta robots and X-Robots-Tag), robots.txt is not noindex, canonicalisation, duplicate content, soft 404, "Crawled - currently not indexed", "Discovered - currently not indexed", no guarantee

What happens during indexing

Once the crawler has fetched a page (lesson 1.2), the search engine reads it. Google analyses the text, the title, the headings, images and videos, and works out which language the page is in. Google also checks whether the page is a copy of another page, and which of those copies is the "real" one. Only then does Google decide whether the page goes into the index.

Google says two things about this that are worth remembering:

Inclusion is not guaranteed. Even if your page meets every requirement, Google may decide not to include it.
It is normal for not everything to be included. In the help for the indexing report, Google literally says you shouldn't expect every address on your site to be indexed, and that this can be perfectly fine. What matters are your important pages.

The three technical requirements

Google lists three technical requirements a page must meet to be eligible for the index:

Googlebot isn't blocked. The page is reachable and robots.txt doesn't forbid fetching it.
The page works. The server returns status code 200, not an error.
The page has indexable content. Text in a file type Google can read, and no violation of the spam policies.

In practice, two choices by the site owner are added to this: a noindex ("don't include this") and a canonical that points to another address ("not this one, but that one"). Together that makes four locks a page has to pass through.

Lock 1: the status code

Only a page with status 200 can get into the index. A 301 or 308 sends the search engine on to the new address; that new address is then assessed. A 404 or 410 removes a previously included address from the index.

The tricky variant is the soft 404 from lesson 1.2: status 200, but the content is an error message or an almost empty page. Google often recognises this and doesn't include such a page. An empty category page in an online shop ("no products found") can also be treated as a soft 404.

Lock 2: robots.txt

With robots.txt you forbid a crawler to fetch a page. That is not the same as forbidding it to include the page. Google is clear about this: robots.txt is not a way to keep a page out of Google. If other sites link to a blocked page, the address can still appear in the results, just without a description. In Search Console this is called "Indexed, though blocked by robots.txt".

Robots.txt is meant for managing crawl traffic: for example, to stop a crawler from working its way through thousands of search results pages on your own site. Lesson 3.3 goes into this in more depth.

Lock 3: noindex

If you really want a page to stay out of the search results, use noindex. There are two ways to do this, with the same effect:

<!-- in the <head> of the page -->
<meta name="robots" content="noindex">

X-Robots-Tag: noindex

The second is an HTTP header that the server sends along. It also works for files that aren't HTML, such as a PDF. The value none means the same as noindex, nofollow. With <meta name="googlebot" content="noindex"> the rule applies to Google only; other search engines may interpret a general rule differently.

Now for the pitfall that comes up very often: noindex and a block in robots.txt don't work together. If robots.txt blocks the page, Google doesn't fetch it, so Google never sees the noindex. The page can then still end up in the index through links from other sites. Want a page out of Google? Let it be fetched and put a noindex on it.

Where might you come across noindex by accident?

In WordPress, under Settings, Read, there is the checkbox "Discourage search engines from indexing this site". This puts a noindex on the whole site. Handy while building, disastrous if you forget to switch it off.
In the AI Builder you can set a site or page to "do not index"; the page then gets a noindex, nofollow. A free demo site is never indexed either.
An SEO plugin or theme with a wrong setting per page or per type of page.

Other website builders have similar switches, often with a name like "hide from search engines".

Lock 4: the canonical

Many sites have the same page under several addresses: with and without www, with http and https, with a parameter added such as ?utm_source=nieuwsbrief. From each such group Google picks one address as the canonical version, and shows only that address in the results. The others are duplicates.

You can help Google choose with a redirect (a strong signal) or with a canonical link in the page:

<link rel="canonical" href="https://www.example.com/bruidsboeketten">

A canonical is a signal, not a command: Google may choose a different address. If your canonical points to another address, the page itself is saying: "don't include me, include that other one". That's fine if it's correct, and a problem if it's a mistake, for example when every page points to the homepage. Lesson 3.5 goes into this further.

According to Google, duplicate content on your own site is not a violation of the spam policies and doesn't lead to a manual action. It is inefficient, though: crawlers fetch the same thing more often, and you don't decide yourself which address appears in the results.

The messages in Search Console

In the Pages report in Search Console (lesson 2.4) you see a reason for each page that isn't indexed. All four locks show up there:

Message (roughly)	Lock	What it means
Not found (404), Soft 404, Server error (5xx), Page with redirect	1	The status code doesn't allow inclusion
Blocked by robots.txt	2	Google wasn't allowed to fetch the page
Excluded by 'noindex' tag	3	Google saw a noindex
Alternate page with proper canonical tag; Duplicate without user-selected canonical	4	Another address was chosen as the canonical version
Crawled - currently not indexed	none	Google has read the page and hasn't included it (yet)
Discovered - currently not indexed	none	Google knows the address but hasn't fetched it yet

The last two aren't technical errors. "Crawled - currently not indexed" is common; Google says it isn't always a problem. If it concerns an important page, check whether it really adds something: a unique, useful page has a better chance than a thin page that looks like ten others.

Summary

After crawling, the search engine decides whether a page goes into the index; that is never guaranteed and not everything needs to be in it.
Four locks: status 200, not blocked in robots.txt, no noindex, canonical absent or pointing to itself.
robots.txt forbids fetching, not inclusion. Use noindex to keep something out of the results, and don't block that page in robots.txt.
A canonical is a strong signal, not a command. Duplicate content isn't spam, but it is inefficient.
"Crawled - currently not indexed" isn't a technical error; look at the value of the page.

Quiz

Exercise

Exercise A1.3: May this page be indexed? Enter the address of a page (a page on your own site, or a page you deliberately hide, such as a thank-you page). Then choose what you expect: "may be indexed" or "may not be indexed", and if you choose "may not", which lock is stopping it (status code, robots.txt, noindex, canonical to another address). The program checks the four locks as described in this lesson and shows its result next to yours.

Cookies at WebOké

Cookie settings

Necessary

Marketing