How to view the raw HTTP request sent by the HttpWebRequest class?

Question

I know that you all will answer "use a debugging proxy such as Fiddler", but it's not that simple.

Here is my situation: I have code that runs on the server in an ASP.NET page with code (aspx.cs) that (by the way) establishes a connection to another server, captures some things, and then formats it and returns it to browser.

The problem is that the other server is doing the wrong thing, and therefore I want to be able to pass the debug flag to the page (via the query string, for example? Debug = true) so that it prints a completely raw HTTP request that it sends to another server, so that I can see what kind of mistake this is. This code works in several places, so I want to be able to simply pass this flag to a dev, intermediate or production process and just see the request, without having to find out if production servers can talk to some proxy server that exists somewhere, etc.

You might think that it would be easy to do this, right? So I feel like I'm crazy or something, but I looked at the link to HttpWebRequest and its parent class WebRequest and nothing. No. You would think that Microsoft would think about it. The closest thing is that you can access the Headers collection, but when I tried it, it omitted some really important headers, such as “content length”, so it should “lie” to me (I know this is lying, because what I know for the fact that the remote server returns the status 200 - the request is successful, it just returns bad / different / wrong data)

Here is the sample code for the request:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://www.whatever.com"); req.Method = ... whatever ...; ... other setup for the request ... /* At this point we are about to send the request. What does the raw HTTP request look like? */ HttpWebResponse resp = (HttpWebResponse)req.GetResponse(); 

Answer 1

You can use the System.Net tracing mechanism to view raw HTTP requests sent to the wire. You can also add your own tracelistener to the process.

Answer 2

I understand that this is an old question. @Feroze's answer says what to do, but does not reveal details about how to configure System.Net tracing to achieve it.

Since this question was Google’s first result for my query on the topic, and since we are all busy people, I thought I would save all of you from having to track down this information.

System.Web very powerful for debugging HttpWebRequest and can be easily configured using web.config :

 <configuration> <system.diagnostics> <trace autoflush="true" /> <sources> <source name="System.Net" maxdatasize="1024"> <listeners> <add name="MyTraceFile"/> <add name="MyConsole"/> </listeners> </source> </sources> <sharedListeners> <add name="MyTraceFile" type="System.Diagnostics.TextWriterTraceListener" initializeData="System.Net.trace.log" /> <add name="MyConsole" type="System.Diagnostics.ConsoleTraceListener" /> </sharedListeners> <switches> <add name="System.Net" value="Verbose" /> </switches> </system.diagnostics> </configuration> 

When you add a simple HttpWebRequest to your code and run in debug mode in Visual Studio, the following information will be displayed in the debug console:

 System.Net Verbose: 0 : [6596] WebRequest::Create(https://example.com/service.asmx) System.Net Verbose: 0 : [6596] HttpWebRequest#62063506::HttpWebRequest(https://example.com/service.asmx#11234) System.Net Information: 0 : [6596] RAS supported: True System.Net Verbose: 0 : [6596] Exiting HttpWebRequest#11234::HttpWebRequest() System.Net Verbose: 0 : [6596] Exiting WebRequest::Create() -> HttpWebRequest#11234 System.Net Verbose: 0 : [6596] HttpWebRequest#11234 ::GetRequestStream() System.Net Verbose: 0 : [6596] ServicePoint#11234 ::ServicePoint(example.com:443) System.Net Information: 0 : [6596] Associating HttpWebRequest#11234with ServicePoint#11234 System.Net Information: 0 : [6596] Associating Connection#11234 with HttpWebRequest#11234 System.Net Information: 0 : [6596] Connection#11234 - Created connection from xxxx:xx to xxxx:xx. System.Net Information: 0 : [6596] TlsStream#11234 ::.ctor(host=example.com, #certs=0) System.Net Information: 0 : [6596] Associating HttpWebRequest#11234 with ConnectStream#11234 System.Net Verbose: 0 : [6596] Exiting HttpWebRequest#11234 ::GetRequestStream() -> ConnectStream#11234 System.Net Verbose: 0 : [6596] ConnectStream#7740977::Write() System.Net Verbose: 0 : [6596] Data from ConnectStream#11234::Write System.Net Verbose: 0 : [6596] 00000000 : 3C 73 6F 61 70 3A 45 6E-76 65 6C 6F 70 65 0D 0A : <soap:Envelope.. ...etc 

I found this especially useful when I was trying to figure out the cause of a web service client error. It turned out I was missing a headline.

Answer 3

You can use a network traffic sniffer like wireshark .

This is not a debugging proxy, but will sniff all traffic and view raw requests / responses.

Answer 4

answering my own question here because I was thinking of another way to do this. Basically the idea is that you redirect the HttpWebRequest to a page that registers the incoming raw HTTP request. In other words, configure a custom HTTP handler according to this forum post:

http://forums.asp.net/t/353955.aspx

And then change only the URL in the HttpWebRequest to point to this new endpoint, but keep all the other request elements the same. Write the result to a file or something else and you are golden.

Answer 5

I suggest you download Telerik Fiddler to capture incoming / outgoing traffic.

Here is a simple example of how to do this with this tool:

• Make sure Capture Traffic is enabled:
• Open a browser and refresh the page or just send a request through an HTTP client.
• After this switch to Fiddler, you should see your request:
• At the top, go to the "Raw" tab.
• The window below is your raw request

Answer 6

Another suggestion. Deploy your own web proxy and ask your request to use it with WebRequest.Proxy . Then you can extract traffic from the proxy instance.

Edit: update links.

Answer 7

You say you think .NET is lying to you, and the specific example you give is that the Content-Length header is not in the HTTP response.

But the Content-Length header is not required from the HTTP response. In fact, if the response body is in any dynamics, and if its length is not known in advance, then it is very likely that the Content-Length header will be omitted!

Answer 8

I know this is an old question, but I found myself in a difficult position when I did not manage the application configuration file, so I needed an easy way to enable tracing through the code and then easily access the raw request / response data in the event. So I put together this custom class, HttpRawTraceListener, which may be useful to others in my position:

https://github.com/jhilgeman/HttpRawTraceListener/blob/master/HttpRawTraceListener.cs

It was designed so that it was easy to add a file to the project, and then call:

 System.Diagnostics.HttpRawTraceListener.Initialize(); 

... to start the trail. From there, requests / responses will be analyzed from trace messages and then will be available through the System.Diagnostics.HttpRawTraceListener.FinishedCommunication event.

It’s probably not 100% ideal for each scenario (for example, it is not a proxy server, so it won’t capture web requests from a browser, for example), but it works pretty well for capturing HttpWebRequests requests / responses to web services, and this can be a good starting point if you need something like that.

Answer 9

I need to miss something because getting an raw HTTP request in ASCII text is very simple, as long as you grab it in Page_Init (), it will be different from Page_Load ().

 protected void Page_Init(object sender, EventArgs e) { //this gets the raw request as an ASCII String. byte[] biData = Request.BinaryRead(Request.TotalBytes); string sWholeRequestAsString = System.Text.Encoding.ASCII.GetString(biData); }