Library recommendations: NodeJs reads a CSV file

Question

Using nodejs, I want to parse a CSV file of 10,000 entries and perform some operations on each line. I tried to use http://www.adaltas.com/projects/node-csv . I could not make it stop in every row. It just reads all 10,000 records. I need to do the following

1. read CSV line by line
2. perform laborious operation on each line
3. go to the next line

Can anyone suggest any alternative ideas here?

Answer 1

It looks like you need to use some kind of streaming solution, such libraries already existed, so before you invent yourself, try this library, which also includes validation support. https://www.npmjs.org/package/fast-csv

Answer 2

My current solution uses an asynchronous module to execute sequentially:

var fs = require('fs'); var parse = require('csv-parse'); var async = require('async'); var inputFile='myfile.csv'; var parser = parse({delimiter: ','}, function (err, data) { async.eachSeries(data, function (line, callback) { // do something with the line doSomething(line).then(function() { // when processing finishes invoke the callback to move to the next one callback(); }); }) }); fs.createReadStream(inputFile).pipe(parser); 

Answer 3

I used this method: -

 var fs = require('fs'); var parse = require('csv-parse'); var csvData=[]; fs.createReadStream(req.file.path) .pipe(parse({delimiter: ':'})) .on('data', function(csvrow) { console.log(csvrow); //do something with csvrow csvData.push(csvrow); }) .on('end',function() { //do something wiht csvData console.log(csvData); }); 

Answer 4

The node -csv project that you link to is sufficient for the task of converting each row of most of the CSV data from documents: http://csv.adaltas.com/transform/ :

 csv() .from('82,Preisner,Zbigniew\n94,Gainsbourg,Serge') .to(console.log) .transform(function(row, index, callback){ process.nextTick(function(){ callback(null, row.reverse()); }); }); 

In my experience, I can say that this is also a fairly quick implementation, I worked with it on data sets with almost 10 thousand records, and the processing time at a reasonable level is tens of milliseconds for the entire set.

A backup proposal based on a yurka-based reduction: node -csv IS stream based and follows the Node.js' API.

Answer 5

To pause streaming in fast-csv, you can do the following:

 let csvstream = csv.fromPath(filePath, { headers: true }) .on("data", function (row) { csvstream.pause(); // do some heavy work // when done resume the stream csvstream.resume(); }) .on("end", function () { console.log("We are done!") }) .on("error", function (error) { console.log(error) }); 

Answer 6

The Fast-CSV npm module can read data line by line from a CSV file.

Here is an example:

 let csv= require('fast-csv'); var stream = fs.createReadStream("my.csv"); csv .fromStream(stream, {headers : true}) .on("data", function(data){ console.log('I am one line of data', data); }) .on("end", function(){ console.log("done"); }); 

Answer 7

• This solution uses csv-parser instead of the csv-parse used in some of the answers above.
• csv-parser appeared about 2 years after csv-parse .
• They both solve the same goal, but personally I found csv-parser better, because it is easy to handle headers through it.

Install csv-parser first:

 npm install csv-parser 

So, suppose you have a CSV file like this:

 NAME, AGE Lionel Messi, 31 Andres Iniesta, 34 

You can perform the necessary operation as:

 const fs = require('fs'); const csv = require('csv-parser'); fs.createReadStream(inputFilePath) .pipe(csv()) .on('data', function(data){ try { console.log("Name is: "+data.NAME); console.log("Age is: "+data.AGE); //perform the operation } catch(err) { //error handler } }) .on('end',function(){ //some final operation }); 

For further reading contact

Answer 8

I needed an asynchronous CSV reader, and I first tried @Pransh Tiwari's answer, but couldn't get it to work with await and util.promisify() . In the end, I came across node-csvtojson , which does the same as csv-parser, but with promises. Here is an example of using csvtojson in action:

 const csvToJson = require('csvtojson'); const processRecipients = async () => { const recipients = await csvToJson({ trim:true }).fromFile('./recipients.csv'); // Code executes after recipients are fully loaded. recipients.forEach((recipient) => { console.log(recipient.name, recipient.email); }); }; 

Answer 9

Try connecting the npm plugin one at a time.

 npm install line-by-line --save 

Answer 10

it is my decision to get the CSV file from an external URL

 const parse = require( 'csv-parse/lib/sync' ); const axios = require( 'axios' ); const readCSV = ( module.exports.readCSV = async ( path ) => { try { const res = await axios( { url: path, method: 'GET', responseType: 'blob' } ); let records = parse( res.data, { columns: true, skip_empty_lines: true } ); return records; } catch ( e ) { console.log( 'err' ); } } ); readCSV('https://urltofilecsv'); 

Answer 11

 fs = require('fs'); fs.readFile('FILENAME WITH PATH','utf8', function(err,content){ if(err){ console.log('error occured ' +JSON.stringify(err)); } console.log('Fileconetent are ' + JSON.stringify(content)); }) 

Answer 12

You can convert CSV to JSON using the CSV-JSON module, and then you can easily use the JSON file in your program

Answer 13

A workaround to solve this problem with await / async :

 const csv = require('csvtojson') const csvFilePath = 'data.csv' const array = await csv().fromFile(csvFilePath); 

Answer 14

NPM install CSV

Example CSV file You will need a CSV file for analysis, so either you already have it, or you can copy the text below and paste it into a new file and call this file "mycsv.csv".

 ABC, 123, Fudge 532, CWE, ICECREAM 8023, POOP, DOGS 441, CHEESE, CARMEL 221, ABC, HOUSE 1 ABC, 123, Fudge 2 532, CWE, ICECREAM 3 8023, POOP, DOGS 4 441, CHEESE, CARMEL 5 221, ABC, HOUSE 

Code reading example and parsing a CSV file

Create a new file and paste the following code into it. Be sure to read what happens behind the scenes.

  var csv = require('csv'); // loads the csv module referenced above. var obj = csv(); // gets the csv module to access the required functionality function MyCSV(Fone, Ftwo, Fthree) { this.FieldOne = Fone; this.FieldTwo = Ftwo; this.FieldThree = Fthree; }; // Define the MyCSV object with parameterized constructor, this will be used for storing the data read from the csv into an array of MyCSV. You will need to define each field as shown above. var MyData = []; // MyData array will contain the data from the CSV file and it will be sent to the clients request over HTTP. obj.from.path('../THEPATHINYOURPROJECT/TOTHE/csv_FILE_YOU_WANT_TO_LOAD.csv').to.array(function (data) { for (var index = 0; index < data.length; index++) { MyData.push(new MyCSV(data[index][0], data[index][1], data[index][2])); } console.log(MyData); }); //Reads the CSV file from the path you specify, and the data is stored in the array we specified using callback function. This function iterates through an array and each line from the CSV file will be pushed as a record to another array called MyData , and logs the data into the console to ensure it worked. var http = require('http'); //Load the http module. var server = http.createServer(function (req, resp) { resp.writeHead(200, { 'content-type': 'application/json' }); resp.end(JSON.stringify(MyData)); }); // Create a webserver with a request listener callback. This will write the response header with the content type as json, and end the response by sending the MyData array in JSON format. server.listen(8080); // Tells the webserver to listen on port 8080(obviously this may be whatever port you want.) 1 var csv = require('csv'); 2 // loads the csv module referenced above. 3​ 4 var obj = csv(); 5 // gets the csv module to access the required functionality 6​ 7 function MyCSV(Fone, Ftwo, Fthree) { 8 this.FieldOne = Fone; 9 this.FieldTwo = Ftwo; 10 this.FieldThree = Fthree; 11 }; 12 // Define the MyCSV object with parameterized constructor, this will be used for storing the data read from the csv into an array of MyCSV. You will need to define each field as shown above. 13​ 14 var MyData = []; 15 // MyData array will contain the data from the CSV file and it will be sent to the clients request over HTTP. 16​ 17 obj.from.path('../THEPATHINYOURPROJECT/TOTHE/csv_FILE_YOU_WANT_TO_LOAD.csv').to.array(function (data) { 18 for (var index = 0; index < data.length; index++) { 19 MyData.push(new MyCSV(data[index][0], data[index][1], data[index][2])); 20 } 21 console.log(MyData); 22 }); 23 //Reads the CSV file from the path you specify, and the data is stored in the array we specified using callback function. This function iterates through an array and each line from the CSV file will be pushed as a record to another array called MyData , and logs the data into the console to ensure it worked. 24​ 25 var http = require('http'); 26 //Load the http module. 27​ 28 var server = http.createServer(function (req, resp) { 29 resp.writeHead(200, { 'content-type': 'application/json' }); 30 resp.end(JSON.stringify(MyData)); 31 }); 32 // Create a webserver with a request listener callback. This will write the response header with the content type as json, and end the response by sending the MyData array in JSON format. 33​ 34 server.listen(8080); 35 // Tells the webserver to listen on port 8080(obviously this may be whatever port you want.) Things to be aware of in your app.js code In lines 7 through 11, we define the function called 'MyCSV' and the field names. If your CSV file has multiple columns make sure you define this correctly to match your file. On line 17 we define the location of the CSV file of which we are loading. Make sure you use the correct path here. 

Launch the application and check the functionality. Open the console and enter the following command:

Node app 1 Node app You should see the following output in your console:

 [ MYCSV { Fieldone: 'ABC', Fieldtwo: '123', Fieldthree: 'Fudge' }, MYCSV { Fieldone: '532', Fieldtwo: 'CWE', Fieldthree: 'ICECREAM' }, MYCSV { Fieldone: '8023', Fieldtwo: 'POOP', Fieldthree: 'DOGS' }, MYCSV { Fieldone: '441', Fieldtwo: 'CHEESE', Fieldthree: 'CARMEL' }, MYCSV { Fieldone: '221', Fieldtwo: 'ABC', Fieldthree: 'HOUSE' }, ] 

1 [MYCSV {Fieldone: 'ABC', Fieldtwo: '123', Fieldthree: 'Fudge'}, 2 MYCSV {Fieldone: '532', Fieldtwo: 'CWE', Fieldthree: 'ICECREAM'}, 3 MYCSV {Fieldone: '8023', Fieldtwo: 'POOP', Fieldthree: 'DOGS'}, 4 MYCSV {Fieldone: '441', Fieldtwo: 'CHEESE', Fieldthree: 'CARMEL'}, 5 MYCSV {Fieldone: '221', Fieldtwo: 'ABC', Fieldthree: 'HOUSE'},] Now you should open a web browser and go to your server. You should see how it outputs data in JSON format.

Conclusion Using node.js and its CSV module, we can quickly and easily read and use the data stored on the server, and make it available to the client upon request.