Fix Indexed Though Blocked by Robots.txt

If you are actively monitoring your Page Indexing report in Google Search Console, you might stumble across a warning that seems like a complete contradiction: "Indexed, though blocked by robots.txt".

​You probably used your robots.txt file to intentionally block search engines from crawling specific URLs, like admin pages, search result pages, or author archives. So, why is Google Search Console telling you that the page was indexed anyway?

​This warning happens because a robots.txt file only stops crawling; it does not stop indexing. Here is exactly why Google bypassed your block and the steps you need to take to properly remove these pages from the search results.

​Key Takeaways

• ​A robots.txt file prevents Google from crawling a page, but not from indexing it.
• ​If Google finds links pointing to your blocked URL, it can still index the page.
• ​Pages indexed this way usually appear in search results without a meta description.
• ​To permanently fix this, you must remove the robots block and use a noindex tag instead.

​Why This Warning Appears

​To understand this warning, you have to remember that search engines separate the actions of reading a page and storing a page.

​When Googlebot visits your website, it checks your robots.txt file first. If you tell the bot to stay away from a specific URL, it will obey and stop crawling. However, if Google finds links pointing to that blocked URL from other websites (or even from internal links on your own blog), it assumes the page is important.

​Because it is not allowed to read the page, Google will index a blind version of it. The search engine will display the page in its search results using only the URL and whatever anchor text other sites used to link to it.

​Quick Note: If you need a refresher on the mechanics of how bots navigate these files, check out our step-by-step breakdown of how Google crawls websites.

​How to Properly Fix the Issue

​Many bloggers make the mistake of leaving the robots.txt block in place, hoping Google will eventually drop the page. To fix this warning, you actually have to do the exact opposite.

​1. Remove the Robots.txt Block

​You cannot tell Google to drop a page from its index if the bot is not allowed to read your instructions. First, you must go into your Blogger settings and edit your custom robots.txt file. Remove the "Disallow" rule for the specific URL causing the warning.

​2. Apply a Noindex Tag

​Now that Googlebot is allowed to visit the page again, you need to give it the correct instructions. You must add a "noindex" meta tag to the HTML header of the affected page. This specific tag explicitly tells search engines that they are forbidden from adding the page to their database.

​3. Request a Recrawl

​Once the noindex tag is live and the robots.txt block is gone, open Google Search Console. Inspect the URL and click "Request Indexing." When the bot visits the page this time, it will read your noindex tag and finally drop the URL from the search results, clearing the warning.

​Important Detail: If you are dealing with other URLs that are strictly blocked and acting normally, you can review our dedicated guide on fixing standard robots.txt blocked errors for comparison.

​Frequently Asked Questions (Q&A)

​Does this warning hurt my website SEO?

It rarely hurts your overall site authority, but it creates a terrible user experience. Because Google cannot read the page, it cannot generate a proper title or description for the search results, making your website look unprofessional to anyone who sees it.

​Can I just delete the page entirely?

Yes. If the page is completely useless and you do not want it to exist at all, you can delete it so it returns a 404 Not Found error. Google will eventually realize the page is gone and deindex it naturally.

​Why shouldn't I use robots.txt for SEO?

Robots.txt is a crawl management tool, not an indexing tool. You should only use it to save server resources. To truly control what shows up in search results, you must rely on canonical tags and noindex directives.