I have a simple program that splits a data set (CSV file) into 4 pieces, reads each fragment, performs some calculations, and then adds the result together. Think of it as a simple map reduction operation. About 1 GB of memory is used to process one block. I run the program on a quad-core PC with 4 GB of RAM running Windows XP. I happen to have encoded it with R, but I don't think this is relevant.
I have encoded two versions. One version processes each fragment in a sequence. Another version processes pieces in two in parallel. Both versions take almost the same amount of time to complete.
Under what circumstances do you expect to see this result?
My current hypothesis is that processes are limited by memory performance, but I don't know how best to investigate this further. Any suggestions or guesses?
Edit: The program is not tied to IO in terms of disk. At the processing stage, a fragment of the CSV file is read into memory, dumped to it for 5 minutes or so, and then the result is written back to the file on disk. File input and output takes only a few seconds.
source
share