NodeJS: upload remote file to S3 with request and write

Question

I am trying to create a way to upload a file from a URL to s3 using a request and knox. Currently my code is as follows:

request(item.productImage, function(err, res, body) { if (!err && res.statusCode == 200) { fs.writeFile('/tmp/' + filename, body, 'base64', function(err, data){ if (err) { return console.log(err); } client.putFile('/tmp/' + filename, '/item/' + item._id + '/' + filename, function(err, res) { if (err) { return console.log(err); } }); }); } }); 

This does not work, since it downloads about 652 bytes of a 4KB file before it stops. Oddly enough, if I don't provide a fs.writeFile () callback, it only loads 4kb locally.

What is the best way to achieve this?

Answer 1

Stackoverflow has a few questions about this, but I cannot find the answer to your question. The solution below should work, however, I am having problems working with knox on my machine right now. Hope you are more lucky!

UPDATE: I seem to have had problems with s3 here, the code below - I really changed one, you need to specify encoding as null for the request, so you get the Buffer back. Otherwise, binary data will not work so well.

 request(item.productImage, {encoding: null}, function(err, res, body) { if(!err && res.statusCode == 200) { var req = client.put('/item/' + item._id + '/' + filename, { 'Content-Type': res.headers['content-type'], 'Content-Length': res.headers['content-length'] }); req.on('response', function(res) { console.log('response from s3, status:', res.statusCode, 'url:', req.url); }); req.on('error', function(err) { console.error('Error uploading to s3:', err); }); req.end(body); } }); 

Note. With this solution, you avoid buffering files to disk - so I decided to use the put method of the lower level client knox.