Truncated Response Using HttpWebRequest

Question

I use the following code to do HttpWebRequests on a website:

public static HttpWebResponse SendGETRequest(string url, string agent, CookieContainer cookieContainer) { HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url); request.UserAgent = agent; request.Method = "GET"; request.ContentType = "text/html"; request.CookieContainer = cookieContainer; return (HttpWebResponse)request.GetResponse(); } 

Everything worked fine with multiple web pages until I tried with a new one and only got the last part of the page. This is the response received:

 <tr> <td colspan="2" height="5"><spacer type="block" width="100%" height="5"></td> </tr> </table> </td> </tr> </table> </body> </html> 

The header is correct and says that only received data is sent. The following are the request and response headers:

Request:

 GET /Broker/Ops/FichaContratoJS.asp?nc=815044&IP=5&YY=2012&M=6&St=0&CC=FESX201206 HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19 Content-Type: text/html Host: www.xxxx.com Cookie: ASPSESSIONIDACBDCDBT=MGDNMNABOANDMILHBNCIDFCH;Autenticacion=Sid=230fae3d%2De0e2%2D4df1%2D8aa8%2D000fb352eaef&IdUsuarioWeb=xxxx; ASPSESSIONIDACBCCDAT=AFDJMNABAFJDDHABLOLAINDK; ASPSESSIONIDCADCBCAT=CEBJGNABLCALPJLDJFPBMLDE 

Answer:

 HTTP/1.1 200 OK Date: Wed, 09 May 2012 07:25:03 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET Pragma: no-cache **Content-Length: 155** Content-Type: text/html Expires: Wed, 09 May 2012 07:24:02 GMT Set-Cookie: Autenticacion=Sid=230fae3d%2De0e2%2D4df1%2D8aa8%2D000fb352eaef&IdUsuarioWeb=xxxx; path=/ Cache-control: no-cache 

Doing the same with a web browser works fine and returns a content length of about 4000 bytes.

Any ideas?

PD: Just in case, it matters. I make several calls to SendGETRequest from different threads to the same site, but since there are no common variables, I think this should not change.

EDIT: this is the extension I use to extract text from a stream:

  public static string ReadTextResponse(this Stream stream) { int count; Encoding enconding = System.Text.Encoding.GetEncoding(1252); System.Text.StringBuilder stringBuilder = new StringBuilder(); byte[] buffer = new byte[1023]; do { count = stream.Read(buffer, 0, buffer.Length); if (count != 0) { string tempString = enconding.GetString(buffer, 0, count); stringBuilder.Append(tempString); } } while (count > 0); return stringBuilder.ToString(); } 

As far as I know, this is correct. Also, note that the response header from the server contains the length of the truncated data.

Answer 1

Incredible ... I sent URL / Broker / Ops / FichaContratoJS.asp? Nc = 815044 & IP = 5 & YY = 2012 & M = 6 and the browser sent /Broker/Ops/FichaContratoJS.asp? ps = 815044 & IP = 5 & YY = 2012 & M = 06 & (pay attention to the additional value 0 by parameter M (month). Putting 0 there, return the full page. It looks like a defect for me

Answer 2

I think you are not using the HttpWebResponse object correctly.

You may not close the request or read the entire strem response.

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse.aspx

Your method should be:

 public static string SendGETRequest(string url, string agent, CookieContainer cookieContainer) { var request = (HttpWebRequest)WebRequest.Create(url); request.UserAgent = agent; request.Method = "GET"; request.ContentType = "text/html"; request.CookieContainer = cookieContainer; string result; using (var myResponse = (HttpWebResponse) request.GetResponse()) { using (var stream = myResponse.GetResponseStream()) { result = null; if (stream != null) { using (var sr = new StreamReader(stream, System.Text.Encoding.UTF8)) { result = sr.ReadToEnd(); sr.Close(); } stream.Close(); } } myResponse.Close(); } return result; } 

Answer 3

I came across a similar situation and found that copying the response stream to a MemoryStream seems to fix my problems.

 public static string SendGETRequest(string url, string agent, CookieContainer cookieContainer) { var request = (HttpWebRequest)WebRequest.Create(url); request.UserAgent = agent; request.Method = "GET"; request.ContentType = "text/html"; request.CookieContainer = cookieContainer; string result; using (var myResponse = (HttpWebResponse) request.GetResponse()) { using (var stream = myResponse.GetResponseStream()) { result = null; if (stream != null) { MemoryStream memStream = new MemoryStream(); stream.CopyTo(memStream); memStream.Flush(); stream.Close(); using (var sr = new StreamReader(memStream, System.Text.Encoding.UTF8)) { result = sr.ReadToEnd(); sr.Close(); } memStream.Close(); } } myResponse.Close(); } return result; }