Why can I login to amazon site using python mechanics but not requests or urllib2

Question

I can use the following python code snippet found from here to log in to amazon.com:

import mechanize br = mechanize.Browser() br.set_handle_robots(False) br.addheaders = [("User-agent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13")] sign_in = br.open('https://www.amazon.com/gp/sign-in.html') br.select_form(name="sign-in") br["email"] = ' [e-mail] ' br["password"] = 'test4test' logged_in = br.submit() orders_html = br.open("https://www.amazon.com/gp/css/history/orders/view.html?orderFilter=year-%s&startAtIndex=1000" % 2013) 

But the following two parts using the query module and urllib2 do not work.

 import requests import sys username = " [e-mail] " password = "test4test" login_data = ({ 'email' : fb_username, 'password' : fb_password, 'flex_password': 'true'}) url = 'https://www.amazon.com/gp/sign-in.html' agent ={'User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.57 Safari/537.1'} session = requests.session(config={'verbose': sys.stderr}, headers = agent) r = session.get('http://www.amazon.com') r1 = session.post(url, data=login_data, cookies=r.cookies) r2 = session.post("https://www.amazon.com/gp/css/history/orders/view.html?orderFilter=year-2013&startAtIndex=1000", cookies = r1.cookies) 

#

 import urllib2 import urllib import cookielib amazon_username = " [e-mail] " amazon_password = "test4test" url = 'https://www.amazon.com/gp/sign-in.html' cookie = cookielib.CookieJar() login_data = urllib.urlencode({'email' : amazon_username, 'password' : amazon_password,}) opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie)) opener.addheaders = [('User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.57 Safari/537.1')] opener.open('www.amazon.com') response = opener.open(url, login_data) response = opener.open("https://www.amazon.com/gp/css/history/orders/view.html?orderFilter=year-%s&startAtIndex=1000" % 2013, login_data) 

What did I do wrong by posting an Amazon registration form? This is the first time I submit a form. Any help is appreciated.

I prefer to use urllib2 or queries because all my other code uses these two modules.

In addition, can any body comment on the speed of work between mechanization, queries and urllib2 and the other advantage of mechanization over the other two?

~~~~~~~~~~~ New ~~~~~~~~~~~~~ Following CC instructions, I can now log in with urllib2. But when I try to do the same with the queries, it still doesn't work. Can someone give me a hint?

 import requests import sys fb_username = " [e-mail] " fb_password = "xxxx" login_data = ({ 'email' : fb_username, 'password' : fb_password, 'action': 'sign-in'}) url = 'https://www.amazon.com/gp/sign-in.html' agent ={'User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.57 Safari/537.1'} session = requests.session(config={'verbose': sys.stderr}, headers = agent) r = session.get(url) r1 = session.post('https://www.amazon.com/gp/flex/sign-in/select.html', data=login_data, cookies=r.cookies) b = r1.text 

Answer 1

As for your urllib2 approach, you are missing two things.

Firstly, if you look at the source of sign-in.html , it shows that

 <form name="sign-in" id="sign-in" action="/gp/flex/sign-in/select.html" method="POST"> 

The form value must be represented by select.html .

Secondly, in addition to email and password, you also need to choose whether you are an existing user or not:

 <input id="newCust" type="radio" name="action" value="new-user"...> ... <input id="returningCust" type="radio" name="action" value="sign-in"...> 

It should look something like this:

 import cookielib import urllib import urllib2 amazon_username = ... amazon_password = ... login_data = urllib.urlencode({'action': 'sign-in', 'email': amazon_username, 'password': amazon_password, }) cookie = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie)) opener.addheaders = [('User-agent', ...)] response = opener.open('https://www.amazon.com/gp/sign-in.html') print(response.getcode()) response = opener.open('https://www.amazon.com/gp/flex/sign-in/select.html', login_data) print(response.getcode()) response = opener.open("https://www.amazon.com/") # it should show that you are logged in print(response.getcode())