Is robots.txt content itself indexed?

Question

The contents of my robots.txt file are actually indexed and displayed in Google search results. This is only Google, not Yahoo, for example.

I really think that Google should not understand in order to index the contents of my robots file, since it is only there to tell Google not to index!

Did I miss something?

Answer 1

This is normal. My country was indexed a few months ago. I think this is a problem with the finder. edit:, you can remove the link on the webmaster tools website.

Answer 2

I would not recommend abandoning the robots.txt file from myself. Instead, you can use the X-Robots-Tag http header with a noindex value, which will prevent search engines from indexing the file.

This will allow them to access your robots.txt file, but will not show it in the search results.

Answer 3

You can add Disallow: /robots.txt to your robots.txt file. I think he should avoid indexing robots.txt.

Answer 4

I think that abandoning your own robots.txt file (if possible) will lead to the fact that all other commands written there will be ignored by bots as well.