Calculate MD5 checksum for file when writing file in C

Question

I have a C application that generates very large binaries, each about 30 GB. After writing each file, it takes some time to calculate the MD5 checksum (after about a couple of minutes per file).

How can I calculate the MD5 checksum of a file as it is being written to disk? I believe that by doing this, I would at least save the extra overhead of reading the file again to calculate the checksum afterwards.

I use the standard C library to execute all IO files, and the OS is Linux.

Can this be done? Thanks!

Answer 1

This can certainly be done. Essentially, you initialize the MD5 calculation before writing. Then, whenever you write some data to disk, also pass this to the MD5 update function. After recording all the data, you call the last MD5 function to calculate the final digest.

If you do not have an MD5 code, RFC 1321 contains a reference implementation of MD5 that provides the above operations.