Access Javascript variable inside HTML file

Question

I am doing a bit of reverse engineering in the Rapportive API in Gmail.

I am making this request

import requests url ='https://api.linkedin.com/uas/js/xdrpc.html' r = requests.get(url) print r.text 

The answer is an empty HTML file with lots of Javascript. On line 3661, he sets RequestHeader for a subsequent Rapportive call:

 ak.setRequestHeader("oauth_token", ae); 

Is there a way I can request this page and then return ae ?

Answer 1

If you are just interested in getting a marker, you cannot just execute the regex:

 var str = '<script>var a = 1;...ak.setRequestHeader("oauth_token", ae);...</script>'; var token = str.match(/setRequestHeader\("oauth_token",\s*([^)]+)/)[1]; 

Although this suggests that ae is the actual string value. If it is a variable, this approach will not work as easily.

Edit: if it is a variable, you can do something like:

 str.replace(/\w+\.setRequestHeader\([^,]+,\s*([^)]+)\s*\);/, 'oauthToken = \1'; 

Before running the JavaScript returned from the page, the global oauthToken (note the missing "var") will contain the value of the token, assuming that the code is evaluated in the same area as the caller.

Answer 2

I think you can try:

• Get the page as you have already done;
• Remove all non-javascript elements in the response page;
• Prepare javascript (described below) on the javascript page to override some code;
• Run it with eval('<code>') ;
• Check if the token is set correctly;

I suggest the following code to override XMLHttpRequest functionality. setRequestHeader to get the token:

 // this will keep the token var headerToken; // create a backup method XMLHttpRequest.prototype.setRequestHeaderBkp = XMLHttpRequest.prototype.setRequestHeader; // override the "setRequestHeader" method XMLHttpRequest.prototype.setRequestHeader = function(key, val) { if ('oauth_token' === key) headerToken = val; this.setRequestHeaderBkp(key, val); }