One in four of a site's crawler-only pages were noindexed anyway

One in four of a site's crawler-only pages were noindexed anyway

A quarter of the pages one of our sites built for search crawlers were pages we'd already told search engines not to list.

Every listing on that site has a claim this listing page beside it, one per listing, and each one gets built fresh from the database on request. The line telling search engines to skip it sits inside the page, so the page is built and sent in full before a crawler can read the line. Only robots.txt stops the request before it costs anything, and that path was in nobody's robots.txt.

In one sample, 19 of 77 freshly built pages were claim pages. We put the path in robots.txt on every site of ours that carries it, and clicking claim this listing in a browser still works.

<caption>Search crawlers kept loading pages we had already told them to ignore, so now we ask them not to visit at all.</caption>

#SEO