I am running a C # console application that is multi-threaded. The main process extracts some data for work, splits it into a configurable amount of smaller data sets, and then generates the same number of threads to process each subset of data.
To process a single record, the thread must make a web service call using the WebRequest class and the POST method. The request is sent using GetRequestStream (), and the response is returned using GetResponse ().
In pseudo-code, the procedure looks something like this:
prepare WebRequest data;
* get time (start-of-Processing);
Stream str = request.GetRequestStream();
Write data to stream;
stream.Close();
WebResponse resp = request.GetResponse();
* get time (response-received);
process response;
finally close response stream;
Temporary data suggests that when we split our data into more than 4 threads, our throughput for the process as a whole does not improve, and in some cases even drops. The synchronization data from the web service remains intact.
- In 4 threads, our obvious overhead is to send data and get the average response stream around the second.
- When we run more than 4 threads, average growth with maximum values colliding with tens of seconds!
Today, I managed to start two separate processes, each of which performed 4 threads (but essentially ensured that each thread still worked with unique data). This time, we almost doubled our total throughput, and each process had a stable time frame of about a second.
, - WebRequest; , . , BeginGetRequestStream BeginGetResponse, , , - ??
, ?