Deny specific URL in robots.txt file

We implemented a rating system on the site, and then a link to the script. Nevertheless, with the overwhelming majority of ratings on the site being 3/5 and ratings very much even at 1-5, we are beginning to suspect that search robots, etc. Pass. The URLs used are as follows:

http://www.thesite.com/path/to/the/page/rate?uid=abcdefghijk&value=3

When we started, add the following to our robots.txt:

User-agent: *
Disallow: /rate

Is it wrong or googlebot and others just ignore our robots.txt?

+3
source share
3 answers

POST , , . , , (, wget), .

, javascript .

robots.txt: , .. http://www.thesite.com/robots.txt - /blah/rate, Disallow: /blah/rate Disallow: /rate

+5

: *
Disallow:/path/to/the/page/rate


.

, : http://www.javascriptkit.com/howto/robots.shtml

0

Looks like me. You restrict access only to http://www.thesite.com/rate(and the pages below this section of IIRC). Plus, some scanners ignore robots.txt!

It is better to make the scores only change in response to POST, not GET. Search engines never use POST.

0
source

Source: https://habr.com/ru/post/1745763/


All Articles