Iterate a large set of files in a directory

Question

I have a directory with 100,000 files, and I need to repeat them to read the value. Right now I am using listFiles() to load all the files into an array and then iterate one by one. But is there an efficient memory way to do this without loading into an array?

 File[] tFiles = new File(Dir).listFiles(); try { for (final File tFile : tFiles) { //Process files one by one } } 

Answer 1

Starting with Java 7, you can use the file visitor template to recursively view the contents of a directory.

The documentation for the FileVisitor interface FileVisitor here .

This allows you to iterate over files without creating a large array of File objects.

A simple example for printing file names:

 Path start = Paths.get(new URI("file:///my/folder/")); Files.walkFileTree(start, new SimpleFileVisitor<Path>() { @Override public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException { System.out.println(file); return FileVisitResult.CONTINUE; } @Override public FileVisitResult postVisitDirectory(Path dir, IOException e) throws IOException { if (e == null) { System.out.println(dir); return FileVisitResult.CONTINUE; } else { // directory iteration failed throw e; } } }); 

Answer 2

Java 8 lazily downloadable stream version:

 Files.list(new File("path to directory").toPath()).forEach(path -> { File file = path.toFile(); //process your file }); 

Answer 3

If you want to avoid the excessive template that comes with the JDK FileVisitor , you can use Guava . Files.fileTreeTraverser() gives you a TreeTraverser<File> , which you can use to move files in a folder (or even subfolders):

 for (File f : Files.fileTreeTraverser() .preOrderTraversal(new File("/parent/folder"))) { // do something with each file }