Checking HTTP response from grunt task

Question

I need to check the accuracy of 4000+ URLs I get from a web service before adding them to a javascript file. I use the grunt task to perform some cleanup operation on these URLs, and I would also like to check that each URL returns 200 HTTP status code before adding them to the js file, therefore as part of the grunt task.

In an example based on the result of the valiate_url task, I need to change the urlToProxy array

For clarity, the whole task I want to build is:

• Read URLs from API and write to file
• Clear the URL list (another task, not sample code)
• Validate URLs to check if server responds with 200
• Write the url of the file containing a simple js array

Any idea / suggestion on how I can do this?

grunt.initConfig({
...

    exec: {
      validate_url: {
        cmd: function(url){
          return 'curl -sL -w "%{http_code}\\n" "http://' + url + '" -o /dev/null';
        },
        callback: function (error, stdout, stderr) {
          if(stdout==200){
            // YES 200 returned
          } else {
            // OOPS NO 200 returned
          }
        }
      }
    }
});

grunt.registerTask('readconfig', 'reads the configuration', function() {

    var urls = grunt.file.readJSON('.tmp/proxyUrls.json');
    var urlsToProxy = urls.record.split('\n');

    for(var i = urlsToProxy.length - 1; i >= 0; i--) {
        grunt.task.run('exec:validate_url:' + 'urlsToProxy[i]);
    }
}

content proxyUrl.js

{ "record": "audentes.com\nfortuna.com\niuvat.com\n...\nwww.google.com" }

Answer 1

You can do this with curl and exec in Grunt.

Below you can find the full sample code. Remember to create package.json. This should be done as follows:

$npm init
$npm install grunt-exec --save
$grunt

It should be in your Gruntfile.js

module
    .exports = function(grunt) {
        grunt
            .loadNpmTasks('grunt-exec'); // register the exec task (from package.json)
        grunt
            .initConfig({
                exec: {
                  validate_url: {
                    cmd: function(url){
                        console
                            .log('validate: ' + url); // log that we are validating
                        return 'curl -sL -w "%{http_code}|%{url_effective}\\n" "http://www.google.com" -o /dev/null -m 5'; // the actual curl. Note the -m 5, this is a 5 second timeout. Also note the -w, this is the writeout which we will split later
                    },
                    callback: function (error, stdout, stderr) {
                        var stdoutSplit = stdout
                                            .split('|');
                        if(stdoutSplit[0]==200){ // 200 status is returned
                            console
                                .log('YES 200, the requested url was: ' + stdoutSplit[1]); // YES 200 returned
                        } else {
                            console
                                .log('Crap, no 200, the requested url was: ' + stdoutSplit[1]);// OOPS NO 200 returned
                        }
                    }
                  }
                }
            });

        grunt
            .registerTask('default', function() {
                grunt
                    .file
                    .readJSON('proxyUrls.json')
                    .record.split('\n')
                    .forEach(function(val){
                        if(val !== ''){
                            grunt.task.run('exec:validate_url:' + val);
                        }
                    });
            });
    };

Note :
Try the command line:curl -sL -w "%{http_code}\\n" "https://pastebin.com/raw.php?i=RPSbdNgR" -o /dev/null