.Google.com's John Mueller answered a concern concerning why Google.com indexes pages that are actually prohibited coming from crawling through robots.txt as well as why the it's safe to dismiss the similar Search Console documents concerning those crawls.Bot Web Traffic To Question Guideline URLs.The person asking the concern recorded that robots were actually making links to non-existent question guideline URLs (? q= xyz) to webpages along with noindex meta tags that are additionally shut out in robots.txt. What motivated the question is actually that Google.com is crawling the hyperlinks to those webpages, acquiring obstructed through robots.txt (without watching a noindex robotics meta tag) after that obtaining turned up in Google.com Explore Console as "Indexed, though obstructed through robots.txt.".The person asked the adhering to concern:." However here's the significant inquiry: why will Google.com mark webpages when they can't also view the web content? What's the perk during that?".Google.com's John Mueller verified that if they can not creep the webpage they can't view the noindex meta tag. He additionally creates a fascinating acknowledgment of the internet site: search operator, urging to ignore the outcomes considering that the "average" customers will not find those outcomes.He composed:." Yes, you're right: if we can't creep the web page, our experts can not find the noindex. That stated, if our company can not crawl the web pages, after that there's certainly not a lot for us to index. Thus while you might see some of those webpages with a targeted web site:- inquiry, the common customer won't find them, so I definitely would not fuss over it. Noindex is likewise alright (without robots.txt disallow), it just indicates the URLs will definitely wind up being actually crawled (and also find yourself in the Look Console report for crawled/not catalogued-- neither of these standings lead to problems to the rest of the website). The essential part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the constraints in using the Web site: hunt advanced hunt driver for diagnostic factors. Among those reasons is due to the fact that it is actually not connected to the routine hunt index, it's a separate thing entirely.Google.com's John Mueller commented on the site search operator in 2021:." The brief response is actually that a web site: concern is certainly not meant to become complete, neither utilized for diagnostics functions.An internet site query is a certain sort of search that restricts the results to a particular website. It's basically just words website, a colon, and afterwards the web site's domain name.This inquiry restricts the results to a particular web site. It is actually not suggested to become a comprehensive collection of all the webpages coming from that web site.".2. Noindex tag without utilizing a robots.txt is alright for these sort of conditions where a crawler is actually connecting to non-existent webpages that are obtaining discovered through Googlebot.3. Links with the noindex tag will produce a "crawled/not catalogued" entry in Explore Console and also those won't have an unfavorable impact on the remainder of the web site.Go through the concern as well as respond to on LinkedIn:.Why will Google index pages when they can't even see the web content?Featured Photo by Shutterstock/Krakenimages. com.