How can I use Java multithreading to read from multiple files?

Question

I want to read and parse many files. Since there are more than 10,000 files that need to be analyzed, I want to make this process faster using streams.

For example, if I had 5 threads, I want them all to read a certain number of files at the same time, so that the reading and parsing process is faster. Is it possible? Can I get significant acceleration by dividing this into threads? If so, how can I do this?

PS I do not mind the use of external libraries.

I work with jdk 1.6

Answer 1

If you have many files to read, the best approach is to have no more than one stream read each file. And the best way to handle many tasks with multiple threads — in most cases — is to use the ExecutorService, which uses a thread pool. Submit a task to the service for each file you read. Make the thread pool large enough to keep the I / O system busy (which could be a bottleneck) and you will maximize performance.

Answer 2

See How to read all lines of a file in parallel in Java 8 to read one file at a time.

In your case, I just launched a thread pool with as many threads as your process allows, each with a request to "read the entire file" for the file assigned to it and let the OS decide which files to read in which order.