Trading CPU RAM (performance issue)

Question

I work with a program that deals with files, I can do several things, for example, rename them, read their contents, etc.

Today I initialize it as follows:

return new Promise((resolve, reject) => { glob("path/for/files/**/*", { nodir: true }, (error, files) => { files = files.map((file) => { // properties like full name, basename, extension, etc. }); resolve(files); }); }); 

So, I read the contents of a specific directory, returned all the files in the array, and then used Array.map to iterate through the array and change the paths for the object with properties.

Sometimes I work with 200,000 text files, so this becomes a problem because it consumes too much RAM.

So, I want to replace with a lazy loading construct .. but I never did this ... so I'm looking for help.

What is my code:

 class File { constructor(path) { this.path = path; } extension() { return path.extname(this.path); } // etc } 

So my main question is: should I only return the property rating, or should I replace it? Like this:

 extension() { this.extension = path.extname(this.path); } 

I understand that this is a compromise. I am going to exchange memory for processor use.

Thanks.

Answer 1

If you want to reduce the use of RAM, I suggest you keep an additional metadata file for each path, as shown below:

• If necessary, save the paths in memory or some of them.

• Saving file properties to hard disk

 files.forEach( (file) => { // collect the properties you want for the file // ... var json = { path: file, extension: extension, .. } // mark the metadata file so you can access it later, for example: put it in the same path with a suffix var metaFile = path + '_meta.json'; fs.writeFile(metaFile, JSON.stringify(json), (err) => { if (err) throw err; }); }); 

Now all the metadata is on the hard drive. So I suppose you are trading memory for disk space and CPU calls.

3. If you want to get the properties for the file, just read the JSON.parse corresponding metadata file.

Answer 2

There is no reason to trade a processor for space. Just walk through the tree and process the files as they are discovered. The space required to walk the tree is proportional to the depth of the tree, if it first made a depth. This almost certainly has the same overhead as just creating a list of paths in your existing code.

For searching directories, the node.js FAQ recommends node-findit . The documentation there is pretty clear. Your code will look something like this:

 var finder = require('findit')(root_directory); var path = require('path'); var basenames = []; finder.on('file', function (file, stat) { basenames.push(path.basename(file)); // etc } 

Or you can wrap the captured values ​​in an object if you want.

Answer 3

If you save only the path property, then the instance of the NodeJS class accepts for your example 200k * (path.length * 2 + 6) byte memory.

If you want to use lazy loading for base names, extenstions, etc., use lazy getters

 class File { constructor(path) { this.path = path; this._basename = null; this._extname = null; } get extname() { return this._extname || (this._extname = path.extname(this.path)); } get basename() { return this._basename || (this._basename = path.basename(this.path)); } }