A document that should be private needs access protection. A document that may be public but should stay out of search needs an indexing decision. Those are different jobs, and robots.txt is often asked to do both. On 2 July, Google announced the retirement of unsupported robots.txt rules, including noindex, from 1 September. Sites relying on that instruction need to choose the right replacement for the real purpose of each page.
Ask who should be able to open the URL
Imagine a supplier sharing a price sheet intended only for approved buyers. If anyone with the URL can open it, a robots rule does not make the document private. The file may be linked, forwarded or discovered in another way. Put the required authentication or other appropriate access control in front of the information itself.
Now imagine a public thank-you page after a newsletter signup. It may not need to appear as a standalone search result, yet there is no secret in its text. That is an indexing question. Using the same technique for both examples hides their different requirements and can give the owner false confidence about the price sheet.
Crawling and indexing need separate instructions
Robots.txt controls crawling by compliant crawlers. If Google cannot fetch a page, it may not see an indexing instruction placed in the page. Google’s announcement lists alternatives including a robots meta tag or HTTP header, removal responses for content that is gone and password protection for private material. The right choice depends on whether the content remains public, is retired or requires authentication.
Have the developer explain the selected response for a representative URL. Inspect the actual delivered page or header, not just a CMS checkbox. If a file such as a PDF needs a header-based instruction, confirm that it applies to the file response. A setting on the HTML page linking to it may not govern the file itself.
Do not mass-edit exclusions without an inventory
Collect the affected rules and paths before replacing them. Some folders may contain old campaign pages, some may hold account functions and others may be needed resources. Assign the intended treatment to each group. Removing all restrictions at once to simplify the configuration can expose pages that were excluded for a sound reason.
Preserve the reason for the decision alongside the rule. A later editor should be able to tell that a page is deliberately public but excluded, rather than assuming it has been forgotten. Keep access-control decisions with the person responsible for the information, not only with whoever edits the robots file.
A search result and an access test answer different questions
After implementing a replacement, verify what an unauthenticated visitor can retrieve and what indexing instruction is delivered. Then monitor search processing through appropriate tools. A result disappearing from search is not proof that the underlying document became private, and a public URL being absent today is no promise it cannot be found tomorrow.
The September change gives teams a clear reason to review old robots settings now. Use that review to resolve the underlying intent. Public, excluded, retired and private are four different states; the website should implement the one the business actually needs.
