WebClient Alternative

Question

I just saw a web crawler in action on my computer and it loads like thousands of metatag info in just a few minutes.

And when I use WebClient to load pages and then analyze them locally, why does it take about 40 seconds for WebClient to load a single web page? Is there an alternative to loading web pages?

thank:)

Answer 1

A few things to consider:

• How many pages do you download at once? Web robots tend to run in parallel.
• .NET . - , . <connectionManagement> - , .
• WireShark, , ? - 40 , , , WebClient.
• , , ?

, API (, WebRequest) , .

Answer 2

, , .

, # , API WebRequest/WebClient . HTTP-, Socket.XxxxAsync, 20 . , , .

Answer 3

, :

• /
• HTML
• , WebClient, .

/ .

Answer 4

, Webclient, - . MSDN Social . , Asyncronous , , , . , , , , .

, Webrequest ( webclient), MB/s 10-20MB, . , .

Answer 5

These settings can help if you experience a slowdown during scanning.

ServicePointManager.DefaultConnectionLimit = int.MaxValue;
ServicePointManager.MaxServicePoints = int.MaxValue;
ServicePointManager.MaxServicePointIdleTime = 0;

Also, be sure to close the HttpWebRequest if you brought the code to the level.

HttpWebResponse.Close();
HttpWebResponse.GetResponseStream().Close();
HttpWebResponse.GetResponseStream().Dispose();

Mike