All geek questions in one place

What is the point of using a robots.txt file on GitHub pages?

I know that the robots.txt file is used to block third-party content index web crawler sites.

However, if the purpose of this file is to delimit the personal area of ​​the site or to protect the private area, the meaning of which is to try to hide the contents using the robots.txt file, if everything is visible in the GitHub repository?

My question extends examples using a custom domain.

Is there any motivation for using the file robots.txtinside GitHub pages? Yes or no? And why?

Alternative 1
In order for the content to remain effectively hidden, then you will need to pay for the website to get a private repository.

+4
source share
1 answer

The goal robots.txtis not to distinguish between private areas, because robots do not even have access to them. Instead, if you have garbage or something else that you do not want to be indexed by search engines or so.

Say for example. I write Flash games for entertainment, and I use GitHub pages so that games check for updates. I have this file hosted on my GHP, the entire contents of which

10579
2.2.3
https://github.com/iBug/SpaceRider/tree/master/SpaceRider%202

It contains three pieces of information: the internal version number of the new version, the display name of the new version, and a download link. Therefore, it is certainly useless when indexing with scanners, so when I have it robots.txt, I would not want to be indexed.

+3
source

Source: https://habr.com/ru/post/1691220/

More articles:


All Articles