Md5 parallel check file

Question

I have a md5sum file containing many lines. I want to use GNU in parallel to speed up the md5sum validation process. In md5sum, when there is no file input, it takes the md5 line from stdin. I tried this:

cat checksums.md5 | parallel md5sum -c {}

But getting this error:

md5sum 445350b414a8031d9dd6b1e68a6f2367 testing.gz: No such file or directory

How can I check md5sum in parallel?

Answer 1

Assuming checksums.md5 has the format:

d41d8cd98f00b204e9800998ecf8427e  My file name

Run:

cat checksums.md5 | parallel --pipe -N1 md5sum -c

If your files are small: -N100

If this does not speed up your processing, make sure your disks are fast enough: md5sum can handle 500 MB / s. iostat -dkx 1can tell you if your drives are a bottleneck.

Answer 2

--pipe. stdin stdin, . man parallel :

cat checksums.md5 | parallel --pipe md5sum -c -

1 , --block.