File processing at the same time

Question

I have a directory in which many files are saved dynamically. Currently, there is a task that from time to time lists files and processes them sequentially (writing to the database). Due to the increase in the number of files, it is necessary to implement parallel processing of these files. Can you give me some ideas and sample code in java, please?

Answer 1

Use the ExecutorService. Create Executors.newFixedThreadExecutor(n); perhaps you can make file processing a single task (or called) and transfer it to a file that you can use to

 ExecutorService service = Executors.newFixedThreadExecutor(10); for(final File file : directory.listFiles()){ service.submit(new Runnable(){ public void run(){ //do work here on file object } }); } 

Answer 2

Take a look at the Watch Servie API in java.nio.file. Here's the documentation and tutorial: http://download.oracle.com/javase/tutorial/essential/io/notification.html

This service allows you to register to change file notifications in a directory. For each notification, you can do any necessary processing. Probably a lot easier than implementing your own thing.

Answer 3

to create a saver extends Thread class and handle file manipulation there (in the run() method)?

---

http://download.oracle.com/javase/tutorial/essential/concurrency/

http://download.oracle.com/javase/7/docs/api/java/lang/Thread.html

Answer 4

This is not entirely obvious if you are familiar with concurrency in Java, so I would start by looking at the Java concurrency Tutorial . This is a good place to start.

Then keep in mind that any object that needs to be accessed by multiple threads must be immutable or synchronized.

After that, you can have a thread pool using ExecutorService and execute multiple threads at the same time.

I know that this is not the same process in essence, but provided that you know how to process files, you can look at the following questions about multithreading in a different context: questions about synchronization in java; when / how / to what extent

Parallel processing in Java; tips required, for example, for Runnanble / Callable interfaces

Answer 5

If I understand correctly your processing of one task from reading to loading in the database. You can break this task down into another character-based task (the central point of the database, the central central or central point). For example, you may have different tasks as follows

• The current task, which selects a file from the directory and passes it to the next task.

• IO Centric - a new task is to read the file and save it in memory, and then move on to the next taks.

• DB centric is a new task for loading data from memory into a database and then clearing the memory.

• IO centric - move the file to another location.

To further improve performance, you can implement task 2, 3, 4 using a thread pool. This will allow you to process many files in parallel. Based on the complexity of the task, you can add or remove any task from the list according to your requirements.