Parallel extensions

Question

I have an application with heavy I / O operations, such as copying files, moving and moving files around the file system, copying to backup servers.

I create this program as single-threaded. It works after 2 minutes.

I built another version of this program with parallel extensions and used Task, which works after almost 2 minutes.

In other words, I did not see a performance gain when using Parallels due to the large I / O.

Do I get the same results if I deploy the application on a blade server?

Blade servers handle IO faster / on multiple channels than my workstation?

No use for Parallels with IO-bound applications?

Answer 1

If all you do is copy or move files throughout the system, then the parallelism provided by TPL will not do you much good. For example, the move does not actually use any processor, it just changes the location of the files in the record structure of the disk directory.

File compression is a completely different story. Here you download data and use the CPU for compression before saving it to disk. You might be able to use a pipeline or parallel loop to load / compress / save data in a more efficient way. Instead of having one thread work on compressing each file, you could use multiple threads for different files.

The following code compresses the load of files sequentially and then in parallel. I get the following points on an i7 920 and with an Intel X25 SSD that compresses 329 JPG images for a total of 800 MB.

Sequence: 39901ms

Parallel: 12404 ms

class Program { static void Main(string[] args) { string[] paths = Directory.GetFiles(@"C:\temp", "*.jpg"); DirectoryInfo di = new DirectoryInfo(@"C:\temp"); Stopwatch sw = new Stopwatch(); sw.Start(); foreach (FileInfo fi in di.GetFiles("*.jpg")) { Compress(fi); } sw.Stop(); Console.WriteLine("Sequential: " + sw.ElapsedMilliseconds); Console.WriteLine("Delete the results files and then rerun..."); Console.ReadKey(); sw.Reset(); sw.Start(); Parallel.ForEach(di.GetFiles("*.jpg"), (fi) => { Compress(fi); }); sw.Stop(); Console.WriteLine("Parallel: " + sw.ElapsedMilliseconds); Console.ReadKey(); } public static void Compress(FileInfo fi) { using (FileStream inFile = fi.OpenRead()) { if ((File.GetAttributes(fi.FullName) & FileAttributes.Hidden) != FileAttributes.Hidden & fi.Extension != ".gz") { using (FileStream outFile = File.Create(fi.FullName + ".gz")) { using (GZipStream Compress = new GZipStream(outFile, CompressionMode.Compress)) { inFile.CopyTo(Compress); } } } } } } 

For compression code, see How to compress files.

Answer 2

If you move files on one physical device, you won’t see much of the benefit of using multiple concurrent I / O requests on the same device. The device is already running several orders of magnitude slower than the processor, so several requests made in parallel will still line up to be processed one by one on the device. Your parallel code is serialized because it all accesses the same device that cannot handle more than one request at a time.

You can see a tiny improvement with parallel code if your disk controller implements “elevators”, “scatter” or other out-of-order operations, but the difference in perpendiculars is relatively small.

Where you should find a more useful primary difference for file input / output when you move files between different physical devices. You should be able to move or copy the file on drive A to another location on drive A, and also copy the file on drive B to drive C. With many physical devices, you do not have all the parallel requests waiting for one device to fill all requests.

You will probably see similar results with network I / O: if everything goes through one network / Ethernet network segment, you won’t understand as much parallelism as if you have several network cards and several network segments to work with.

Answer 3

I think the benefit of concurrent extensions can be significant for CPU operations. Donna, how should he influence IO tho.

Answer 4

It all depends on whether you are tied to the CPU or tied to IO. I would suggest doing some performance tests to see where your bottles are.

If you find that you are moving and compressing a large number of files (to different disks, since moving to the same disk is just a change to the FAT table), you may need to implement a stream file engine that shrinks when you move it. This can save extra IO to read files again after moving them. I did this with moving and checksum, and in my case it was a huge performance hit.

Hope this helps.

Answer 5

I have an application that is implemented in WinForms, which processes ~ 7800 URLs in about 5 minutes (loads the URL, analyzes the content, searches for specific pieces of data and, if it finds that it is looking for it, does some additional processing of this data .

This is a specific application, used to run from 26 to 30 minutes, but by changing the code to TPL (parallel task library in .NET v4.0), it only runs 5. The computer is a Dell T7500 workstation with dual-core quad-core Xeon processors (3 GHz ) working with 24 GB of RAM and the 64-bit version of Windows 7 Ultimate.

Although this is not quite the same as your situation, it is too intense. The TPL documentation claims that it was originally designed for problem sets with a processor set, but that does not preclude its use in I / O situations (as my application demonstrates to me). If you have at least 4 cores and you have not noticed that the processing time has been significantly reduced, you may have other implementation problems that prevent TPL from working effectively (locks, hard disk elements, etc.). The book "Parallel Programming with Microsoft.NET" really helped me understand the "how" your code needs to be modified to really use all of these features.

Worth a look, in my opinion.