Does Google not allow web clients?

Question

I have the following:

string html_string = "http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=pharma"; string html; html = new WebClient().DownloadString(html_string); 

and when I get the length of the HTML, it only returns the first 28435 characters.

Is it possible that Google does not allow access to the web client?

Answer 1

No see TOS

5.3. You agree not to access (or attempt to access) any Services by any means other than through the interface provided by Google, unless you are permitted to do so in a separate agreement with Google. You specifically agree not to gain access (or try to gain access) to any of the Services using any automated means (including the use of scripts or web scanners) and must comply with the instructions set forth in any robots.txt file present in the Services.

Answer 2

I tried this snippet and it returned exactly the same HTML that was returned by the browser. The only correction I would make is to dispose of disposable objects:

 string html_string = "http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=pharma"; using (var client = new WebClient()) { string html = client.DownloadString(html_string); } 

Answer 3

If you write a bot, this will not work; they will eventually block you.

You might want to look at their list of APIs , especially Custom Search , and see if this helps?

Answer 4

From experience for search results, they can and will close you; they discover a robot.

Answer 5

This, of course, will be different. The browser may have many additional characters that deal with the registered user, bot code and many other scripts.

When you retrieve data through code, the search will be performed as a user who is not a google user (or a non-signed user, if you want). This is the simplest explanation.

I'm afraid Darin's answer will not work, at least not all the time. This is not perfect.


Yes, of course, your activity will be discovered as a bot, not a person. So beware of the consequences.