robots.txt and noindex: why can a blocked URL still appear?

A useful troubleshooting scenario: a site blocks a utility URL in robots.txt, adds noindex to its HTML, and then expects the URL to disappear from search immediately.

There are two separate controls here. A crawling restriction controls fetching; noindex is an indexing directive the crawler needs to discover. Blocking the page can prevent that discovery. Google's noindex documentation explains the distinction.

My practical checklist would be: confirm the current response, inspect the robots rules, inspect the HTML or HTTP indexing directive, and check the page's links and sitemap inclusion. Only then choose a removal approach.

How would you explain this to someone who thinks robots.txt is a privacy feature?

I would use “do not fetch this” and “do not list this” as separate labels, then add a third label for access control.

A private report needs authentication. Neither of the first two labels prevents a visitor who already knows the address from requesting it. That distinction helps keep a search-management conversation separate from a permissions problem.

I would also separate “the configuration is correct now” from “the search result has changed.” Those are different observations.

For a review log, record the deployment time, the response checked, and the inspection result. A screenshot of an old search result by itself does not tell you whether the crawler has seen the new directive.

Good distinction. I would finish the handoff with an owner and a verification step for each control: engineering checks access and responses; the SEO reviewer checks crawl/index behaviour.

For a public utility page, I would avoid describing noindex as secrecy. For a genuinely private resource, the first question is whether an unauthenticated request can see the data at all.