How to check if a page is noindex
A page that is set to noindex will be dropped from Google whatever else you do to it. It is the single cheapest thing to check, and the most commonly missed, because nothing about the page looks wrong in a browser.
The short answer. Open the page, view source, and search for noindex. If it appears in a <meta name="robots"> tag, the page is blocked. But that check alone misses two of the four ways a page gets blocked, which is why the rest of this page exists.
1. The robots meta tag
The usual method. In the page source, look inside <head> for:
<meta name="robots" content="noindex, nofollow">To check by hand: right click the page, choose View page source, then press Ctrl+F (Cmd+F on a Mac) and search for robots.
Two things to watch. The directive can be addressed to one crawler rather than all of them, in which case the name attribute changes:
<meta name="googlebot" content="noindex">And the absence of a robots meta tag is not a problem. No tag means the default, which is indexable. You are looking for the presence of noindex, not the absence of the tag.
What the directives actually mean
| Directive | Effect |
|---|---|
noindex | Keep the page out of the index entirely. This is the one that costs you rankings. |
nofollow | Do not follow links out of this page. Does not affect whether the page itself is indexed. |
none | Shorthand for noindex, nofollow. Easy to miss because the word "noindex" never appears. |
noarchive | No cached copy. Harmless for rankings. |
index, follow | The default. Writing it out does nothing, good or bad. |
Because none exists, searching the source for the literal string "noindex" can return nothing on a page that is still blocked. Search for robots instead and read the whole tag.
2. The X-Robots-Tag HTTP header
This is the one people miss. The same directives can be sent as a response header rather than a tag, which means the page source will look completely clean while the page is still blocked.
It is common on PDFs, images and other non-HTML files, where there is no <head> to put a tag in, and on servers configured at the CDN or reverse proxy level.
To check it from a terminal:
curl -sI https://example.com/page | grep -i x-robots-tagOr in Chrome DevTools: open the Network tab, reload the page, click the first request in the list (the document itself, not an asset), and read Response Headers.
If a page is missing from Google and the source looks fine, check this header before you check anything else. A header-level noindex is invisible to every check that only reads the HTML.
3. The rendered DOM, not the raw HTML
View source shows you the HTML the server sent. It does not show you what the page became after JavaScript ran. A tag manager, a consent script or a framework can inject or remove a robots meta tag after load, and Google renders pages before indexing them, so the rendered state is the one that counts.
To see the rendered version, open DevTools with F12, go to Elements, and read the <head> there. That panel shows the live DOM. If the two disagree, trust the rendered one.
In practice this cuts both ways: a page can look blocked in source and be fine, or look fine in source and be blocked.
4. robots.txt is not noindex
This confusion is worth its own section because acting on it makes things worse.
A Disallow rule in robots.txt asks crawlers not to fetch the page. It does not ask them not to index it. A disallowed URL can still appear in search results, usually with no description, if other pages link to it.
Worse, the two block each other. If a page is disallowed in robots.txt, Google cannot fetch it, which means it cannot see the noindex tag on it either. The instruction never gets read.
So if you want a page out of the index: allow it in robots.txt, and serve noindex. Blocking it in robots.txt is the way to keep it in.
Why pages end up noindex by accident
- A staging flag shipped to production. The most common cause by a distance. WordPress has a "Discourage search engines" checkbox in Settings, Reading, that writes
noindexsite wide. It survives a migration from a staging site. - An SEO plugin default. Yoast, RankMath and AIOSEO all set
noindexon some archive, tag or author templates out of the box. - A CDN or framework rule that adds
X-Robots-Tagto a path pattern that grew to include real pages. - A per-page toggle someone set during a draft and never cleared.
All four look identical to a visitor. That is the whole problem: a blocked page renders perfectly.
Checking it in one click
Teardown is a free Chrome extension that reads the rendered page in your active tab and opens with the indexability verdict: whether the page is indexable, blocked, or quietly canonicalised somewhere else. It reads the robots meta tag, the canonical, hreflang and the page language together, so you get the answer without four separate checks.
Because it reads the tab you are already looking at, it works on staging sites and pages behind a login, which a third party crawler cannot reach.
Related
- Free heading structure checker, which also reports the robots meta tag for a URL you paste
- How to check the heading structure of a web page
- What Teardown pulls from a page