BlockDex registry proof pages ask crawlers not to index, cost stays uncached

BlockDex registry proof pages ask crawlers not to index, cost stays uncached

Every registry listed on BlockDex has a page where whoever runs that registry can prove it's theirs, and that page asks search engines not to index it. Nobody was ever going to reach it from a search, so none of what it costs to serve comes back.

That instruction sits inside the page. A crawler reads it only after the page has been built and sent, and this page reads the database twice per request and draws the whole site frame around the result. All of that is paid before the line saying don't index this is visible.

So I moved the ask into robots.txt, which crawlers read before they fetch anything. For the ones that ignore robots.txt, BlockDex answers a claim address with 410 rather than 403 or 404, because 410 is the status that makes a crawler drop an address instead of retrying it. Googlebot, Bingbot, Applebot, and the crawlers behind ChatGPT, Claude and Perplexity are excluded from that refusal. A person clicking through to claim a registry still gets the page.

https://blockdex.kynth.studio/li/t28o2f

#shadcn