Is there a way to speed up the opening and hashing of 15,000 small files in C #?

Question

I am working on a SHA1 checksum of 15,000 images (40 KB - 1.0 MB each, just 1.8 GB). I would like to speed it up, as this will be a key operation in my program, and now it takes from 500 to 600 seconds.

I tried the following, which took 500 seconds:

 public string GetChecksum(string filePath)
        {
            FileStream fs = new FileStream(filePath, FileMode.Open);
            using (SHA1Managed sha1 = new SHA1Managed())
            {
                return BitConverter.ToString(sha1.ComputeHash(fs));
            }

        }

Then I thought that maybe the pieces of SHA1Managed () were too small, so I used BufferedReader and increased the size of the buffer to the size of any of the files that I read.

 public string GetChecksum(string filePath)
        {
            using (var bs = new BufferedStream(File.OpenRead(filePath), 1200000))
            {
                using (SHA1Managed sha1 = new SHA1Managed())
                {
                    return BitConverter.ToString(sha1.ComputeHash(bs));
                }
            }
        }

It took 600 seconds.

Is there anything I can do to speed up these I / O operations, or am I stuck with what I got?

---

x0n . , IO, 480 .

Answer 1

SHA1Managed ; . ComputeHash 15 000 , (IMO.)

public Dictionary<string,string> GetChecksums(string[] filePaths)
{ 
    var checksums = new Dictionary<string,string>(filePaths.length);

    using (SHA1Managed sha1 = new SHA1Managed()) 
    { 
         foreach (string filePath in filePaths) {
              using (var fs = File.OpenRead(filePath)) {
                  checksums.Add(filePath, BitConverter.ToString(sha1.ComputeHash(fs)));
              }
         }         
    }
    return checksums;
}

SHA1Managed , /, p/invoke win32.

-Oisin

Answer 2

.

dotTrace: http://www.jetbrains.com/profiler/

Answer 3

, IO.

, . , , (100% ) SHA. IO, .

CPU, ( , 2 , , 2002 ), Sha1Managed() SHA. - 2x. 2- ( ), 4x.

, , , "" , 50% - Windows.

, , . .

Answer 4

"ramdisk" - .

Answer 5

SHA1CryptoServiceProvider SHA1Managed? SHA1CryptoServiceProvider , , . :

public static byte[] CreateSHA1Hash(string filePath)
{
    byte[] hash = null;



    using (SHA1CryptoServiceProvider sha1 = new SHA1CryptoServiceProvider())
    {
        using(FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 131072))
        {
            hash = sha1.ComputeHash(fs);
        }

        //hash = sha1.ComputeHash(File.OpenRead(filePath));
    }

    return hash;
}

, 15000 ( WinAPI: FindFirstFile(), FindNextFile()), .NET Directory.GetFiles().

Directory.GetFiles . , WinAPI.