All geek questions in one place

Priority Question Robots.txt

If I have these lines in a robots.txt file:

Disallow /folder/ Allow /folder/filename.php

Will filename.php be resolved?

In what order does Google prioritize strings?

What will be here, for example ?:

 Allow / Disallow /

Basically, I mean Google here as a search engine.

thanks

+4
source share
2 answers

For your first example, yes, the file /folder/filename.php will be allowed, since Google will prioritize according to the most specific rule. See here or check out the Wikipedia article , for example, for more details.

In addition, Google has detailed documentation on how it uses robots.txt over here .

Edit: For the second example, Allow will be prioritized according to the citation from Wikipedia Byron Whitlock published in his answer.

+5
source

It depends on the search engine:

While the standard implementation of the first matching robots.txt template always wins, Google’s introduction is different in that allowing templates with equal or more characters to the directory path defeats the Deny template. Bing uses Allow or Deny the most specific action.

To be compatible with all robots, if someone wants to allow a single file otherwise the directory is prohibited, you must first place the allow directive (s), then by prohibition.

http://en.wikipedia.org/wiki/Robots_exclusion_standard

+2
source

Source: https://habr.com/ru/post/1334143/

More articles:


All Articles