Where to put the robots.txt file?

Question

Where should the robots.txt file be placed?

domainname.com/robots.txt 

or

 domainname/public_html/robots.txt 

I placed the file in domainname.com/robots.txt , but it does not open when I type it in the browser.

alt text http://shup.com/Shup/358900/11056202047-My-Desktop.png

Answer 1

If the file is located on your file system, it depends on which host you are using, so it’s difficult for us to give a specific answer.

Best description: put it wherever the index.html file (or index.php or whatever) that represents your home page is located. If it is domainname/public_html/index.html , for example, put it in domainname/public_html/robots.txt .

Answer 2

I think the best way to describe this is to have it in the root folder of your domain ... so http://example.com/robots.txt you can also put your sitemap.xml file in the root directory or link to it using Sitemap: http://example.com/fldr/smap.xml in your robots.txt file.

don't forget: you can use Google’s Webmaster Tools to make sure you don’t restrict anything you didn’t want (you can also see woohoo requests and links!).

: id consider using <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> if possible, because you will still earn linkjuice for links on the page, but it won’t appear in the googles index at that time as a robots.txt directive can leave a simple url with a do description in SERP, but lose all the link to it because its robots.txted out (its b / c rating of anchor text to get credit for it)

Answer 3

At the root of your web directory (where you put the files that appear on your website)

Answer 4

In this case, you should put it in domainname/public_html/robots.txt , since the public.html folder will contain your index file.