Swift Calculate MD5 checksum for large files

Question

I am working on creating an MD5 checksum for large video files. I am currently using the code:

extension NSData { func MD5() -> NSString { let digestLength = Int(CC_MD5_DIGEST_LENGTH) let md5Buffer = UnsafeMutablePointer<CUnsignedChar>.allocate(capacity: digestLength) CC_MD5(bytes, CC_LONG(length), md5Buffer) let output = NSMutableString(capacity: Int(CC_MD5_DIGEST_LENGTH * 2)) for i in 0..<digestLength { output.appendFormat("%02x", md5Buffer[i]) } return NSString(format: output) } } 

But this creates a memory buffer, and for large video files it will not be ideal. Is there a way in Swift to calculate the MD5 checksum reading the file stream, so the amount of memory will be minimal?

Answer 1

You can calculate the MD5 checksum in blocks, as shown, for example, in the section Is there an MD5 library that does not require all the input at once? ,

Here is a possible implementation using Swift (now updated for Swift 5)

 import CommonCrypto func md5File(url: URL) -> Data? { let bufferSize = 1024 * 1024 do { // Open file for reading: let file = try FileHandle(forReadingFrom: url) defer { file.closeFile() } // Create and initialize MD5 context: var context = CC_MD5_CTX() CC_MD5_Init(&context) // Read up to 'bufferSize' bytes, until EOF is reached, and update MD5 context: while autoreleasepool(invoking: { let data = file.readData(ofLength: bufferSize) if data.count > 0 { data.withUnsafeBytes { _ = CC_MD5_Update(&context, $0.baseAddress, numericCast(data.count)) } return true // Continue } else { return false // End of file } }) { } // Compute the MD5 digest: var digest: [UInt8] = Array(repeating: 0, count: Int(CC_MD5_DIGEST_LENGTH)) _ = CC_MD5_Final(&digest, &context) return Data(digest) } catch { print("Cannot open file:", error.localizedDescription) return nil } } 

Autoplay of the pool is necessary to free the memory returned by file.readData() , without it the entire (potentially huge) file will be loaded into memory. Thanks to Abhi Beckert for noticing this and providing an implementation.

If you need a digest as a hexadecimal string, change the return type to String? and replace

 return digest 

from

 let hexDigest = digest.map { String(format: "%02hhx", $0) }.joined() return hexDigest 

Answer 2

Solution (based on Martin R's answer) for SHA256 hash:

 func sha256(url: URL) -> Data? { do { let bufferSize = 1024 * 1024 // Open file for reading: let file = try FileHandle(forReadingFrom: url) defer { file.closeFile() } // Create and initialize SHA256 context: var context = CC_SHA256_CTX() CC_SHA256_Init(&context) // Read up to `bufferSize` bytes, until EOF is reached, and update SHA256 context: while autoreleasepool(invoking: { // Read up to `bufferSize` bytes let data = file.readData(ofLength: bufferSize) if data.count > 0 { data.withUnsafeBytes { _ = CC_SHA256_Update(&context, $0, numericCast(data.count)) } // Continue return true } else { // End of file return false } }) { } // Compute the SHA256 digest: var digest = Data(count: Int(CC_SHA256_DIGEST_LENGTH)) digest.withUnsafeMutableBytes { _ = CC_SHA256_Final($0, &context) } return digest } catch { print(error) return nil } } 

Using an instance of the URL type named fileURL previously created:

 if let digestData = sha256(url: fileURL) { let calculatedHash = digestData.map { String(format: "%02hhx", $0) }.joined() DDLogDebug(calculatedHash) }