WebClient prohibits opening a page on Wikipedia?

Question

Here is the code I'm trying to run:

var wc = new WebClient(); var stream = wc.OpenRead( "http://en.wikipedia.org/wiki/List_of_communities_in_New_Brunswick"); 

But I keep getting the forbidden error 403. I don’t understand why. It worked great for other pages. I can open the page in my browser. How can i fix this?

Answer 1

I would usually not use OpenRead() , try DownloadData() or DownloadString() .

It may also be that Wikipedia intentionally blocks your request because you did not specify a user agent string:

 WebClient client = new WebClient(); client.Headers.Add("user-agent", "Mozilla/5.0 (Windows; Windows NT 5.1; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4"); 

I often use WebClient and quickly found out that websites can and will block your request if you do not provide a user agent string that matches a well-known web browser. In addition, if you create your own user agent string (for example, “my super clean web scraper”), you will also be blocked.

[change]

I changed the line of my user agent to the version of the modern version of Firefox. The original example I gave was a user agent string for IE6, which is not a good idea. What for? Some websites can filter based on IE6 and send a message to someone with this browser or to another page that says: “Please refresh your browser” - this means that you won’t receive the content you want to receive.