HttpWebRequest.BeginGetRequestStream () best practice

Question

I am working on an async Http crawler that collects data from various services, and currently I am working with file pools that execute consecutive HttpWebRequest requests to send / receive data from services.

I want to switch to asynchronous web calls (BeginGetRequestStream and BeginGetResponse), I need to somehow get the response data and POST statistics (% completed by recording when it is completed (when it will be more important), etc.). Currently, I have an event that is being called from an object that spawns / contains a stream, and an HTTP data transfer message is being sent. Is there an event in WebRequests that I can connect to to call an already implemented event? This would be the easiest to transition.

Thanks for any help!

Answer 1

You can pass the delegate (as part of the async status parameter) that you want to call. Then after your EndGetResponseStream does what you need, then call this delegate with any parameters you need.

Personally, as you move on to the aysnc programming model (I assume to improve performance), I highly recommend that you move your workflow to asynchronous. This model allows you to process the results as they become available and as quickly as possible without any locks.

Edit

There is an article on my blog

HttpWebRequest - Asynchronous Programming Model /Task.Factory.FromAsyc

. , , , , . GetAsync PostAsync , .

public static void GetAsyncTask(string url, Action<HttpWebRequestCallbackState> responseCallback,
  string contentType = "application/x-www-form-urlencoded")

responseCallback? , .

, ( PostAsyn()

    var iterations = 100;
    for (int i = 0; i < iterations; i++)
    {
      var postParameters = new NameValueCollection();
      postParameters.Add("data", i.ToString());
      HttpSocket.PostAsync(url, postParameters, callbackState =>
        {
          if (callbackState.Exception != null)
            throw callbackState.Exception;
          Console.WriteLine(HttpSocket.GetResponseText(callbackState.ResponseStream));
        });
    }

URL-. GET - (POST), - , , . , , , " ".

Action<HttpWebRequestCallbackState>

HttpWebRequestCallbackState - , , , . Action.

Answer 2

, / ( ) -. , , , .. , , , .

private void ScanSites ()
{
  // for each URL in the collection...
  WebRequest request = HttpWebRequest.Create(uri);

  // RequestState is a custom class to pass info
  RequestState state = new RequestState(request, data);

  IAsyncResult result = request.BeginGetResponse(
    new AsyncCallback(UpdateItem),state);
}

private void UpdateItem (IAsyncResult result)
{
  // grab the custom state object
  RequestState state = (RequestState)result.AsyncState;
  WebRequest request = (WebRequest)state.request;
  // get the Response
  HttpWebResponse response =
    (HttpWebResponse )request.EndGetResponse(result);

  // fire the event that notifies the UI that data has been retrieved...
}

, RequestState , .

, , , , . , , .

Answer 3

System.Net.WebClient:

var client = new WebClient();
client.DownloadDataCompleted += (s, args) => { /* do stuff here */ };
client.DownloadDataAsync(new Uri("http://someuri.com/"));

Answer 4

The second method is my main way to stop responding.

            public string GetResponse()
            {
                // Get the original response.
                var response = _request.GetResponse();

                Status = ((HttpWebResponse) response).StatusDescription;

                // Get the stream containing all content returned by the requested server.
                _dataStream = response.GetResponseStream();

                // Open the stream using a StreamReader for easy access.
                var reader = new StreamReader(_dataStream);

                // Read the content fully up to the end.
                var responseFromServer = reader.ReadToEnd();

                // Clean up the streams.
                reader.Close();
                if (_dataStream != null) 
                    _dataStream.Close();

                response.Close();

                return responseFromServer;
            }

            /// <summary>
            /// Custom timeout on responses
            /// </summary>
            /// <param name="millisec"></param>
            /// <returns></returns>
            public string GetResponse(int millisec)
            {
                //Spin off a new thread that safe for an ASP.NET application pool.
                var responseFromServer = "";
                var resetEvent = new ManualResetEvent(false);
                ThreadPool.QueueUserWorkItem(arg =>
                    {
                        try
                        {
                            responseFromServer = GetResponse();
                        }
                        catch (Exception ex)
                        {
                            throw ex;
                        }
                        finally
                        {
                            resetEvent.Set();//end of thread
                        }
                    });

                //handle a timeout with a asp.net thread safe method 
                WaitHandle.WaitAll(new WaitHandle[] { resetEvent }, millisec);
                return responseFromServer;
            }