JavaScript Command Line URL

Question

I use curl in php and httplib2 in python to get the url.

However, there are some pages that use JavaScript (AJAX) to retrieve data after you load the page, and after that they simply overwrite a specific section of the page.

So, is there a command line utility that can handle JavaScript?

To find out what I mean, go to: monster.com and try to find a job.

You will see that after this Ajax receives a list of tasks. So, if I wanted to complete tasks based on a keyword search, I would get a page without any tasks.

But through the browser it works.

Answer 1

Get FireBug and see the URL for this Ajax request. Then you can use curl with this url.

Answer 2

There are two ways to handle this. Write your screen scraper using a full browser such as Webkit , or go to the actual page and find out what the AJAX request is doing and request it directly. Then you need to analyze the course results. Use firebug to help you.

. . Javascript?

Answer 3

, env.js <script>. Rhino JavaScript interpreter XMLHttpRequest, ( <script>, .js eval), . , , Java-.

John Resig, , , Google.

Answer 4

, HtmlUnit ?

HtmlUnit - " GUI-Less Java-". HTML API, , , .. , "" .

JavaScript ( ) AJAX, Firefox Internet Explorer , .

-.

Answer 5

LiveHttpHeaders Firefox, URL, cURL URL-. LiveHttpHeaders , ​​ ( ), .. , .

Answer 6

you can use PhantomJS http://phantomjs.org

You can use it as below:

var page=require("webpage");
page.open("http://monster.com",function(status){
  page.evaluate(function(){
    /* your javascript code here 
        $.ajax("....",function(result){


            phantom.exit(0);
           }); */
  });
});