Do not allow Google to crawl my sites

Question

Is there any way to hide my drupal site from googlebots? I do not want my site to appear in search engines while it is still under development.

Answer 1

Use a robots.txt file, for example:

User-agent: * Disallow: <directory you want to hide> 

You can allow some directories to be searched if you want, or you can’t use Deny at all: /

Additional information at robotstxt.org: -)

Dave

Answer 2

Use robots.txt .

Answer 3

Use robots.txt to block search engines ...

But!

You need to think about creating your site away from the real domain. When you start living, how are you going to develop improvements to your site? Now you need a development environment.

In addition, you can create your external links and search engine rankings before posting your website live. Preventing your Google site from being indexed until you go live will stop this.

I would suggest placing a landing page in your main domain, using a messaging service that reflects the purpose of your site, and the possibility of possibly “registering interest” on the site. This will allow you to get your domain name.

The launch of a new website will almost always happen with loud silence. Getting your domain rankings before launch is a very rewarding activity.

Answer 4

Create a file containing the following in the base directory of your web server called robots.txt

 User-agent: * Disallow: / 

Answer 5

Another option is to put something like this in you .htaccess

 Order Deny,Allow Deny From all Allow From 1.2.3.4 

and just use your IP address or multiple IP addresses if the site has multiple remote locations.

But I am making a second comment about developing the main domain. Get a VPS account, host the splash site in the primary domain with a virtual host, and create a subdomain and virtual host for it (and protect it as shown above) and use it for development.

Answer 6

Require username. I don’t think the Google bot will log in ...