How to log in to Google using the Python Requests module?

Question

I would like to interact with Google Trends using the Python Requests module. Google Trends requires authentication to access Google Trends data. Is there a way to programmatically log in with requests? It doesn't seem like OAuth 2.0 will work for me since I am not using the Google API.

Answer 1

I think you really get more interesting data by capturing the raw JSON that it uses for plotting. It includes related headers that do not come with CSV downloads. This works for multiple queries (5?) Before you reach the quota.

import re import requests _GOOGLE_TRENDS_URL = 'http://www.google.com/trends/trendsReport?hl=en-US&content=1&q=%s&hl=en-US&content=1' term = 'foo' response = requests.get(_GOOGLE_TRENDS_URL % term) if response.status_code == requests.codes.ok: data_line = [l for l in response.content.splitlines() if 'var chartData' in l][0] chart_data = re.sub(r'.*var chartData = (.*?);.*', r'\1', data_line) # Fix for date representation chart_data = re.sub(r'new Date\((\d+), (\d+), (\d+)\)', r'"\1-\2-\3"', chart_data) data = json.loads(chart_data) #data = { # ..., # "rows": [ # [ # { # "f": "January 2004", # "v": "2004-0-16" # }, # Date # null, # annotation # null, # annotation text # 91, # count # null, # annotation (2?) # null, # annotationText (2?) # true # certainty # ], #... # ] #} for row in data['rows']: if '2013' in row[0]['v']: print '%s: %d' % (row[0]['f'], row[3]) else: print response.status_code print response.text 

Productivity:

 January 2013: 21 February 2013: 21 March 2013: 21 April 2013: 20 May 2013: 20 June 2013: 20 July 2013: 20 August 2013: 21 September 2013: 19 October 2013: 20 November 2013: 21 December 2013 (partial data): 22