How to stream reading a directory in node.js?

Question

Suppose I have a directory containing 100K + or even 500k + files. I want to read the directory with fs.readdir , but it does not update the stream. Someone will tell me that asynchronous memory usage is before reading the entire list of files.

So what is the solution? I want to read with a streaming approach. Can I?

Answer 1

In modern computers, moving a directory with 500K files is nothing. When you fs.readdir asynchronously in Node.js, what it does is just read the list of file names in the specified directory. It does not read the contents of files. I just tested 700K files in a directory. Only 21 MB of memory is required to download this list of file names.

Once you have downloaded this list of file names, you simply go through them one at a time or in parallel, setting a limit for concurrency, and you can easily use them. Example:

 var async = require('async'), fs = require('fs'), path = require('path'), parentDir = '/home/user'; async.waterfall([ function (cb) { fs.readdir(parentDir, cb); }, function (files, cb) { // `files` is just an array of file names, not full path. // Consume 10 files in parallel. async.eachLimit(files, 10, function (filename, done) { var filePath = path.join(parentDir, filename); // Do with this files whatever you want. // Then don't forget to call `done()`. done(); }, cb); } ], function (err) { err && console.trace(err); console.log('Done'); });