Automatically logging in to Google Web Crawler

Question

I would like to automatically detect Google and other crawlers and write them to my ASP.NET site. Has anyone found a reliable way to do this? Part of the entrance is easy, but reliably detecting them is a real problem.

Sincerely.

Answer 1

Take a look at user-agent. It must be googlebot. A more reliable way is to reverse lookup IP addresses. It will tell you whether it is a Google bot or not. I would use both methods. BUT NOTE: you will slow down your site as you will do a reverse search for each visitor.

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=80553

Answer 2

, - , Google , , , - , -.

-, User Agent. , IP-, , Google IP-, .

DNS- , googlebot.com, Googlebot, , .

Answer 3

. . , , Googlebot IP-. , .

Answer 4

Google , , noarchive. , .

. - Google.

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156412