How to implement multi-threaded pool in Java

Question

I have a scenario in which I use streams.

Firstly, I have a folder where there are files that are updated frequently. So, I wrote a stream that reads the contents of the folder and writes the file names to a static list and updates the list if new files appear.

Secondly, I wrote another thread that takes the file names from the list and does some processing with the files.

These two threads work continuously, one checks for new files, processes new files.

Now I need to process three files simultaneously with three threads. When one thread completes processing, another thread takes a different file name from the list and starts the process.

So, I need some kind of mechanism to have three threads and check them, whether they are alive or not, and accordingly launches a new thread, and the list of files is also often updated.

I also looked ExecutorService, but while the list was updated, I could not provide an updated list.

Thanks Sandeep

Answer 1

Based on existing answers, your code will look something like this:

    final ExecutorService executor = Executors.newFixedThreadPool(2);

    final FilePoller poller = ...
    final FileProcessor processor = ...

    new Thread(new Runnable() 
      { 
        @Override
        public void run() 
        {
          while (true)
          {
            final File file = poller.pollForFile();
            executor.submit(new Runnable() { public void run() { processor.processFile(file); } } );
          }
        }
      });

Assuming your processors can keep up with the poller, that would be nice, otherwise you would need to set up a throttling mechanism before sending it to the performer.

Answer 2

; java.util.concurrent.ThreadPoolExecutor Runnable, , , .

Answer 3

Similar to @SimonC suggestion, but instead of a very long comment, I have an answer.

final FilePoller poller = ...
final FileProcessor processor = ...

final ExecutorService executor = Executors.newFixedThreadPool(4);

executor.submit(new Runnable() { 
    public void run() {
        final File file = poller.pollForFile();
        executor.submit(new Runnable() { public void run() { processor.processFile(file); } } );
        // repeat, but wait for a free thread.
        executor.submit(this);
    }
  });
 // to stop the whole thing
 executor.shutdown();

Answer 4

How to monitor changes in a folder and spawn a stream / file, assuming that changing notifications gives you a list of changes in a folder?