Does googlebot crawl urls in jQuery $ .get () calls and can it be prevented?

Question

I have a page that takes the form of this ajaxForm jQuery plugin . The form is submitted, and when it completes, there is a call using $ .get () to load new content onto the page.

My problem is that Googlebot "appears" to index the URL in the $ .get () method.

My first question is: is this possible? I was impressed that Googlebot did not evaluate javascript for the most part (I read something about how it can index content by URLs with! #).

My second question is: if Google indexes this call to this URL, is there a way to prevent it?

Thanks in advance.

Answer 1

You can robots.txtspecify the file googlebot will be to read it.

From robotstxt.org :

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

You can also see the Webmaster Center to remove a file from the list.

Answer 2

First of all, you need to check that it is really GoogleBot, because anyone can pretend to be GoogleBot, even a legitimate user.

Recommended method: do a reverse DNS lookup, make sure the name is on googlebot.com domain, and then do the appropriate DNS-> IP forwarding using this googlebot.com name.

Google Webmaster Central: Googlebot.

Answer 3

googlebot inline-javascript URL-, "/" extenstion ( ".html", ".php" )... .

confuscate URL- JS, . i.e: "/" "|" JS, "|" "/" .

, , , : js , .

robots.txt is not really a solution. because the urls are still being discovered, dragged to the open (the google pipe uses to determine what to scan next), but then the blocking is blocked, which is basically one missed opportunity.