.Google.com's John Mueller responded to an inquiry concerning why Google indexes webpages that are actually prohibited from crawling through robots.txt as well as why the it's risk-free to ignore the associated Browse Console reports about those crawls.Bot Traffic To Question Specification URLs.The individual inquiring the concern documented that robots were actually producing links to non-existent concern parameter URLs (? q= xyz) to webpages with noindex meta tags that are actually also blocked out in robots.txt. What motivated the question is that Google.com is actually creeping the links to those web pages, obtaining shut out by robots.txt (without watching a noindex robotics meta tag) after that obtaining reported in Google Search Console as "Indexed, though obstructed by robots.txt.".The individual inquired the adhering to inquiry:." However right here's the significant question: why would Google.com index webpages when they can't also observe the information? What is actually the benefit during that?".Google's John Mueller affirmed that if they can't creep the web page they can not observe the noindex meta tag. He also creates an appealing reference of the web site: search driver, suggesting to overlook the results given that the "average" users won't see those end results.He wrote:." Yes, you are actually right: if our team can't creep the web page, our company can't observe the noindex. That stated, if our experts can not creep the web pages, then there is actually not a whole lot for us to index. So while you may find a number of those pages with a targeted web site:- question, the typical consumer won't observe all of them, so I wouldn't bother it. Noindex is additionally fine (without robots.txt disallow), it simply implies the Links will end up being actually crept (as well as find yourself in the Browse Console record for crawled/not indexed-- neither of these conditions trigger concerns to the remainder of the internet site). The important part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the constraints being used the Web site: hunt evolved search operator for analysis explanations. One of those reasons is actually since it is actually not connected to the frequent search index, it's a distinct trait altogether.Google.com's John Mueller discussed the web site search driver in 2021:." The brief response is that a site: question is not meant to be full, neither used for diagnostics reasons.An internet site inquiry is a details sort of hunt that restricts the end results to a certain website. It's basically only the word internet site, a colon, and afterwards the internet site's domain name.This concern restricts the end results to a details internet site. It is actually certainly not suggested to become a detailed compilation of all the webpages from that internet site.".2. Noindex tag without using a robots.txt is actually great for these sort of conditions where a robot is actually linking to non-existent web pages that are actually receiving found through Googlebot.3. Links along with the noindex tag will create a "crawled/not recorded" entry in Look Console and also those will not have a negative result on the remainder of the website.Read through the concern and also address on LinkedIn:.Why would Google.com mark web pages when they can not also see the web content?Featured Photo by Shutterstock/Krakenimages. com.