WebClient.DownloadFile () in Parallel.ForEach

Question

This code works fine:

Parallel.ForEach(photos, item => { WebClient webClient = new WebClient(); webClient.DownloadFile(item.src_big, "C:\\pic" + item.ID + ".jpg"); }); 

So far, this code is generating "An exception occurred during a WebClient request."

 foreach (Photo p in photos) { Task.Factory.StartNew(() => { WebClient webClient = new WebClient(); webClient.DownloadFile(p.src_big, "C:\\pic" + p.ID + ".jpg"); }); } 

I had two questions:

1) In the first code, I use several WebClient objects to load. The same goes for the second code, so why am I getting an exception?

2) I try these two versions to determine what is the fastest way to upload photos, in my case from facebook. I would like to know if there is another method that is faster, perhaps WebRequest.Create ()?

Answer 1

You close the loop variable in the second case - try the following:

 foreach (Photo p in photos) { Photo photo = p; Task.Factory.StartNew(() => { WebClient webClient = new WebClient(); webClient.DownloadFile(photo.src_big, "C:\\pic" + photo.ID + ".jpg"); }); } 

Also Parallel.ForEach() is synchronous - after its execution all files have been downloaded. Tasks, on the other hand, can still go on, so you have to wait for them to complete, perhaps something like this would be more suitable for the second case:

 var tasks = photos.Select( p => Task.Factory.StartNew(() => { using(WebClient webClient = new WebClient()) webClient.DownloadFile(p.src_big, string.Format(@"C:\pic{0}.jpg",p.ID)); })).ToArray(); Task.WaitAll(tasks); 

As you can see, Parallel.ForEach() is very preferable in this case, since the syntax is very short, they both use a thread pool under the hood, so choose the simplest option that you can get rid of, especially since you do not need additional complexity.

Also, I don’t think that you get your data faster with WebRequest - most of the delay will be caused by the network / Internet, and not from the two that you choose - that if I were to choose a simpler code, which definitely uses WebClient .

To summarize: I would go for Parallel.ForEach() with WebClient , option 1 .

Answer 2

Asnwer to 2 is that I use the same procedure and it works parallel for me perfectly.foreach significantly reduced the time

When I analyzed the statics, I noticed that if I have a large number of images to receive, then parallel.foreach starts so many streams that it almost reaches the width of my broadband access, for example, I have a maximum regular download speed of 100 kbps so uploading photos reached 100 KB, which means that it could not be better than this, because it is the Internet that matters