All geek questions in one place

How to check if googlebot will index a given URL?

We are making a whitelabel site that should not be indexed by Google.

Does anyone know a tool to check if googlebot will index a given URL?

I placed it <meta name="robots" content="noindex" />on all pages, so it should not be indexed, however I would prefer to check it at 110%.

I know that I can use robots.txt, but the problem with robots.txt is as follows: Our main site should be indexed, and this is the same application on IIS (ASP.Net) as a whitelabel site - the only difference is URL

I cannot modify the robots.txt file depending on the incoming URL, but I can add a meta tag to all pages from my code.

+3
source share
2 answers

Google’s webmaster tools ( google.com/webmasters/tools ) will (besides permission to download the Sitemap) perform a test crawl of your site and say that they crawl, how it evaluates certain requests and what they will crawl and what not .

Crawl scans are not automatically included in Google’s results, anyway, if you try to hide sensitive data from Google’s prying eyes, you can’t only rely on this: put some authentication on the line of fire, no matter what.

+3
source

You must add Robots.txt to your website.

- . ( Robots.txt)

EDIT: Robots.txt .
IIS Robots.txt ASP.Net; IIS.

+4

Source: https://habr.com/ru/post/1744502/

More articles:


All Articles