C # HttpClient.SendAsync always returns 404, but the url works in the browser

Question

I am developing a C # console application to validate a URL. It works well for most urls. But we found that in some cases, the application always received a 404 response from the target site, but the URLs do work in the browser. And these URLs also work when I tried them in tools like DHC (Dev HTTP Client).

At first, though this may be the reason for not adding the correct headers. But after trying to use Fiddler to compose an HTTP request with the same headers, it works in Fiddler.

So what about my code? Is there a bug in the .NET HttpClient?

Here is the simplified code for my test application:

class Program { static void Main(string[] args) { var urlTester = new UrlTester("http://www.hffa.it/short-master-programs/fashion-photography"); Console.WriteLine("Test is started"); Task.WhenAll(urlTester.RunTestAsync()); Console.WriteLine("Test is stoped"); Console.ReadKey(); } public class UrlTester { private HttpClient _httpClient; private string _url; public UrlTester(string url) { _httpClient = new HttpClient { Timeout = TimeSpan.FromMinutes(1) }; // Add headers _httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"); _httpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip,deflate,sdch"); _httpClient.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"); _httpClient.DefaultRequestHeaders.Add("Accept-Language", "sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4"); _url = url; } public async Task RunTestAsync() { var httpRequestMsg = new HttpRequestMessage(HttpMethod.Get, _url); try { using (var response = await _httpClient.SendAsync(httpRequestMsg, HttpCompletionOption.ResponseHeadersRead)) { Console.WriteLine("Response: {0}", response.StatusCode); } } catch (HttpRequestException e) { Console.WriteLine(e.InnerException.Message); } } } } 

Answer 1

This is apparently a problem with accepted languages. I got a 200 response using the following Accept-Language header value

 _httpClient.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ru;q=0.4"); 

ps I assume that you know that in your example _client should read _httpClient in the urlTester constructor or not create it.

Answer 2

Another possible cause of this problem is that the URL you are sending is around 2048 bytes. At this point, the content (almost certainly the query string) may become truncated, and this, in turn, means that it may not be correctly mapped to the server-side route.

Although these URLs were handled correctly in the browser, they were also unable to use the get command in the shell.

This problem was resolved by using POST with key value pairs instead of using GET with a long query string.