WatchService performance with many directories

Question

I want to use the Java WatchService to listen for changes in a large number of directories (many hundreds of thousands), but I do not know if this is suitable for so many directories viewed.

Does anyone have experience with WatchService with so many directories?

If this helps, WatchService will be used on CentOS 6.5 with the EXT4 file system.

Thank you Michael

Answer 1

This situation is quite common for the IDE. They often use directory browsing for complex directory structures and many tens of thousands of files.

Two things can be noted:

• On Linux, you often have to configure the OS to monitor a large number of files. https://confluence.jetbrains.com/display/IDEADEV/Inotify+Watches+Limit

To prevent this situation, it is recommended to increase the clock limit (for example, 512K). You can do this by adding the following line to the /etc/sysctl.conf file:

  fs.inotify.max_user_watches = 524288 

In this example, the system tracks 512k files.

• If you have a hard drive, it won’t make it spin faster, and most likely it’s 80 to 120 IOPS (IO per second), and this is likely to be a performance bottleneck than you might like.

Like many I / O operations in Java, it is a wrapper around an object that is actually implemented by the OS.