CreateReadStream not working / extremely slow for large files

Question

Im using DropBox API to upload files. To upload files to Dropbox, I go through the following steps:

• First upload the file from form to a local directory on the server.
• Reading a file from a local directory using fs.createReadStream
• Send the Dropbox file via the Dropbox API.

Problem:

For some reason, fs.createReadStream takes an absolute age when reading and downloading a large file. Now the file I'm trying to download is 12 MB , which is not a large file, but it takes about 18 minutes to download / process a 12 MB file .

I do not know where the problem is either in createReadStream or in the Dropbox api code.

It works with files of size within kb .

My code is:

 let options = { method: 'POST', uri: 'https://content.dropboxapi.com/2/files/upload', headers: { 'Authorization': 'Bearer TOKEN HERE', 'Dropbox-API-Arg': "{\"path\": \"/test/" + req.file.originalname + "\",\"mode\": \"overwrite\",\"autorename\": true,\"mute\": false}", 'Content-Type': 'application/octet-stream' }, // I think the issue is here. body: fs.createReadStream(`uploads/${req.file.originalname}`) }; rp(options) .then(() => { return _deleteLocalFile(req.file.originalname) }) .then(() => { return _generateShareableLink(req.file.originalname) }) .then((shareableLink) => { sendJsonResponse(res, 200, shareableLink) }) .catch(function (err) { sendJsonResponse(res, 500, err) }); 

Update:

 const rp = require('request-promise-native'); 

Answer 1

I had an experience similar to this question before and after a lot of scratches and digging my head, I was able to solve the problem anyway.

For me, the problem arose because of the default lock size for createReadStream (), which was pretty small 64kb, and this somehow worked when loading into Dropbox.

Thus, the solution was to increase the size of the piece.

 // Try using chunks of 256kb body: fs.createReadStream(`uploads/${req.file.originalname}`, {highWaterMark : 256 * 1024}); 

Answer 2

https://github.com/request/request#streaming

I believe that you need to pass the stream to the request.

see this answer: Sending big image data via HTTP in Node.js