How to limit the response to the length of the content of a simplified HTTP request in node?

Question

I would like to configure a simplified HTTP request () package to interrupt the loading of too large HTTP resources.

Suppose request () is configured to load a URL and the resource size is 5 gigabytes. I want request () to stop loading after 10 MB. Usually, when a request receives a response, it receives all the HTTP headers and everything is behind. As soon as you manipulate the data, you already have all the downloaded data.

There is a maxContentLength parameter in axios, but I cannot find anything similar for request ().

I should also mention that I'm not mistaken, but I load at least the headers and the beginning of the resource.

Answer 1

const request = require('request'); const URL = 'http://de.releases.ubuntu.com/xenial/ubuntu-16.04.3-desktop-amd64.iso'; const MAX_SIZE = 10 * 1024 * 1024 // 10MB , maximum size to download let total_bytes_read = 0; 

1 - If the response from the server is compressed by gzip, you must enable the gzip option. https://github.com/request/request#examples For backward compatibility, response compression is not supported by default. To accept gzip-compressed responses, set the gzip parameter to true.

 request .get({ uri: URL, gzip: true }) .on('error', function (error) { //TODO: error handling console.error('ERROR::', error); }) .on('data', function (data) { // decompressed data console.log('Decompressed chunck Recived:' + data.length, ': Total downloaded:', total_bytes_read) total_bytes_read += data.length; if (total_bytes_read >= MAX_SIZE) { //TODO: handle exceeds max size event console.error("Request exceeds max size."); throw new Error('Request exceeds max size'); //stop } }) .on('response', function (response) { response.on('data', function (chunk) { //compressed data console.log('Compressed chunck Recived:' + chunk.length, ': Total downloaded:', total_bytes_read) }); }) .on('end', function () { console.log('Request completed! Total size downloaded:', total_bytes_read) }); 

NB: If the server does not compress the response, but you still use the gzip / unzip option, then the decompression fragment and the original fragment will be equal. Therefore, you can do a Limit check anyway (from an unpacked / compressed piece) However, if the response is compressed you should check the size limit of the unpacked fragment

2 - if the answer is not compressed, you do not need the gzip parameter to unpack

 request .get(URL) .on('error', function (error) { //TODO: error handling console.error('ERROR::', error); }) .on('response', function (response) { response.on('data', function (chunk) { //compressed data console.log('Recived chunck:' + chunk.length, ': Total downloaded:', total_bytes_read) total_bytes_read += chunk.length; if (total_bytes_read >= MAX_SIZE) { //TODO: handle exceeds max size event console.error("Request as it exceds max size:") throw new Error('Request as it exceds max size'); } console.log("..."); }); }) .on('end', function () { console.log('Request completed! Total size downloaded:', total_bytes_read) }); 

Answer 2

You can also use the data event in this case of the request package. I tested below and it worked fine for me.

 var request = require("request"); var size = 0; const MAX_SIZE = 200; request .get('http://google.com/') .on('data', function(buffer){ // decompressed data as it is received size += buffer.length; if (size > MAX_SIZE) { console.log("Aborting this request as it exceeds max size") this.abort(); } console.log("data coming"); }).on('end', function() { console.log('ending request') }) .on('response', function (response) { console.log(response.statusCode) // 200 console.log(response.headers['content-type']) // 'image/png' response.on('data', function (data) { // compressed data as it is received console.log('received ' + data.length + ' bytes of compressed data') // you can size and abort here also if you want. }) }); 

There are two places where you can perform a size check, where you get compressed data or where you get uncompressed data (based on the above example https://www.npmjs.com/package/request )

Answer 3

As @Jackthomson noted in the response of the first comment, this can be done using .on(data) If you want the headers you can take them from the response, and you can also check the content-length header and not start the separation.

From the certificate of axioms.

// maxContentLength determines the maximum response size HTTP content allowed maxContentLength: 2000,

Here's how the maxContentLength axioms maxContentLength

 var responseBuffer = []; stream.on('data', function handleStreamData(chunk) { responseBuffer.push(chunk); // make sure the content length is not over the maxContentLength if specified if (config.maxContentLength > -1 && Buffer.concat(responseBuffer).length > config.maxContentLength) { reject(createError('maxContentLength size of ' + config.maxContentLength + ' exceeded', config, null, lastRequest)); } }); 

partial request equivalent

 var request = require("request"); const MAX_CONTENT_LENGTH = 10000000; var receivedLength = 0; var req = request.get('http://de.releases.ubuntu.com/xenial/ubuntu-16.04.3-desktop-amd64.iso') .on('response', (response) => { if (response.headers['content-length'] && response.headers['content-length'] > MAX_CONTENT_LENGTH) { console.log("max content-length exceeded") req.abort(); } }) .on('data', (str) => { receivedLength += str.length; if (receivedLength > MAX_CONTENT_LENGTH) { console.log("max content-length exceeded") req.abort(); } })