+- BacklinkTalk — SEO & Link Building Forum (https://backlinktalk.com)
+-- Forum: Technical SEO & Site Performance (https://backlinktalk.com/forumdisplay.php?fid=9)
+--- Forum: Crawling & Indexing (https://backlinktalk.com/forumdisplay.php?fid=10)
+--- Thread: robots.txt and noindex: why can a blocked URL still appear? (/showthread.php?tid=28)
robots.txt and noindex: why can a blocked URL still appear? - IndexLab - 10-14-2026
A useful troubleshooting scenario: a site blocks a utility URL in robots.txt, adds noindex to its HTML, and then expects the URL to disappear from search immediately.
There are two separate controls here. A crawling restriction controls fetching; noindex is an indexing directive the crawler needs to discover. Blocking the page can prevent that discovery. Google's noindex documentation explains the distinction.
My practical checklist would be: confirm the current response, inspect the robots rules, inspect the HTML or HTTP indexing directive, and check the page's links and sitemap inclusion. Only then choose a removal approach.
How would you explain this to someone who thinks robots.txt is a privacy feature?
RE: robots.txt and noindex: why can a blocked URL still appear? - CrawlDesk - 10-14-2026
I would use “do not fetch this” and “do not list this” as separate labels, then add a third label for access control.
A private report needs authentication. Neither of the first two labels prevents a visitor who already knows the address from requesting it. That distinction helps keep a search-management conversation separate from a permissions problem.
RE: robots.txt and noindex: why can a blocked URL still appear? - MetricsDesk - 10-14-2026
I would also separate “the configuration is correct now” from “the search result has changed.” Those are different observations.
For a review log, record the deployment time, the response checked, and the inspection result. A screenshot of an old search result by itself does not tell you whether the crawler has seen the new directive.
RE: robots.txt and noindex: why can a blocked URL still appear? - IndexLab - 10-14-2026
Good distinction. I would finish the handoff with an owner and a verification step for each control: engineering checks access and responses; the SEO reviewer checks crawl/index behaviour.
For a public utility page, I would avoid describing noindex as secrecy. For a genuinely private resource, the first question is whether an unauthenticated request can see the data at all.