All geek questions in one place

How do I send a URL request to a server using Python without actually opening my browser ("without using a web browser module")?

I want to send this URL as a request to the server in order to change something on the website during my login. The problem is that when I use mechanize or urllib2 to open the URL, it doesn’t change anything on the website, However, when I use the webbrowser module, it changes things on the website. I want to do what the webbrowser module does, but WITHOUT opening the actual browser. Is there any way to do this? And why don't mechanization and urllib2 work?

EDIT: What I mean by “website changes” is that I get these things, called “Promotions” and “Tickets”, for the information I posted on the website. My program finds the exact information (they will throw you out if it is fake) and using the URL, it “inserts” it into the site.

Example URL (all my rest follow this format):

http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp $ JspView $ SaveAction & inPlaceID = 1020634218 & xxx_c_1_f_987 = http% 3A% 2F% 2Fwww.yellowpages.com% 2Fpittsburgh-% 2Fmily- 2% dollar-store-1349194% 3Flid% 3D1349194

mechanization code:

import mechanize br = mechanize.Browser() url = http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194 br.open(url)

Urllib2 code:

 from urllib2 import urlopen url = http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194 page = urllib2.urlopen(url) page.read()

web browser code:

 import webbrowser url = http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194 webbrowser.open(url)

EDIT # 2 I tried this code just now:

 import urllib2 import urllib def log_in(): url = 'https://www.locationary.com/index.jsp?ACTION_TOKEN=tile_loginBar_jsp$JspView$LoginAction' values = {'inUserName' : ' me@gmail.com ', 'inUserPass' : 'myPass'} data = urllib.urlencode(values) req = urllib2.Request(url, data) req.add_header('Host', 'www.locationary.com') req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0') req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8') req.add_header('Accept-Language', 'en-us,en;q=0.5') req.add_header('Accept-Encoding','gzip, deflate') req.add_header('Accept-Charset','ISO-8859-1,utf-8;q=0.7,*;q=0.7') req.add_header('Connection','keep-alive') req.add_header('Referer','http://www.locationary.com/') req.add_header('Cookie','site_version=REGULAR; __utma=47547066.1079503560.1321924193.1322707232.1324693472.36; __utmz=47547066.1321924193.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); nickname=jacob501; locaCountry=1033; locaState=1795; locaCity=Montreal; jforumUserId=1; PMS=1; TurnOFfTips=true; Locacookie=enable; __utma=47547066.1079503560.1321924193.1322707232.1324693472.36; __utmz=47547066.1321924193.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); nickname=jacob501; PMS=1; __utmb=47547066.15.10.1324693472; __utmc=47547066; JSESSIONID=DC7F5AB08264A51FBCDB836393CB16E7; PSESSIONID=28b334905ab6305f7a7fe051e83857bc280af1a9; __utmc=47547066; __utmb=47547066.15.10.1324693472; ACTION_RESULT_CODE=ACTION_RESULT_FAIL; ACTION_ERROR_TEXT=java.lang.NullPointerException') req.add_header('Content-Type','application/x-www-form-urlencoded') response = urllib2.urlopen(req) page = response.read() url2 = 'http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194' log_in() response2 = urllib2.urlopen(url2) page2 = response2.read() ; nickname = jacob501; PMS = import urllib2 import urllib def log_in(): url = 'https://www.locationary.com/index.jsp?ACTION_TOKEN=tile_loginBar_jsp$JspView$LoginAction' values = {'inUserName' : ' me@gmail.com ', 'inUserPass' : 'myPass'} data = urllib.urlencode(values) req = urllib2.Request(url, data) req.add_header('Host', 'www.locationary.com') req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0') req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8') req.add_header('Accept-Language', 'en-us,en;q=0.5') req.add_header('Accept-Encoding','gzip, deflate') req.add_header('Accept-Charset','ISO-8859-1,utf-8;q=0.7,*;q=0.7') req.add_header('Connection','keep-alive') req.add_header('Referer','http://www.locationary.com/') req.add_header('Cookie','site_version=REGULAR; __utma=47547066.1079503560.1321924193.1322707232.1324693472.36; __utmz=47547066.1321924193.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); nickname=jacob501; locaCountry=1033; locaState=1795; locaCity=Montreal; jforumUserId=1; PMS=1; TurnOFfTips=true; Locacookie=enable; __utma=47547066.1079503560.1321924193.1322707232.1324693472.36; __utmz=47547066.1321924193.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); nickname=jacob501; PMS=1; __utmb=47547066.15.10.1324693472; __utmc=47547066; JSESSIONID=DC7F5AB08264A51FBCDB836393CB16E7; PSESSIONID=28b334905ab6305f7a7fe051e83857bc280af1a9; __utmc=47547066; __utmb=47547066.15.10.1324693472; ACTION_RESULT_CODE=ACTION_RESULT_FAIL; ACTION_ERROR_TEXT=java.lang.NullPointerException') req.add_header('Content-Type','application/x-www-form-urlencoded') response = urllib2.urlopen(req) page = response.read() url2 = 'http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194' log_in() response2 = urllib2.urlopen(url2) page2 = response2.read() ; __utmc = import urllib2 import urllib def log_in(): url = 'https://www.locationary.com/index.jsp?ACTION_TOKEN=tile_loginBar_jsp$JspView$LoginAction' values = {'inUserName' : ' me@gmail.com ', 'inUserPass' : 'myPass'} data = urllib.urlencode(values) req = urllib2.Request(url, data) req.add_header('Host', 'www.locationary.com') req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0') req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8') req.add_header('Accept-Language', 'en-us,en;q=0.5') req.add_header('Accept-Encoding','gzip, deflate') req.add_header('Accept-Charset','ISO-8859-1,utf-8;q=0.7,*;q=0.7') req.add_header('Connection','keep-alive') req.add_header('Referer','http://www.locationary.com/') req.add_header('Cookie','site_version=REGULAR; __utma=47547066.1079503560.1321924193.1322707232.1324693472.36; __utmz=47547066.1321924193.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); nickname=jacob501; locaCountry=1033; locaState=1795; locaCity=Montreal; jforumUserId=1; PMS=1; TurnOFfTips=true; Locacookie=enable; __utma=47547066.1079503560.1321924193.1322707232.1324693472.36; __utmz=47547066.1321924193.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); nickname=jacob501; PMS=1; __utmb=47547066.15.10.1324693472; __utmc=47547066; JSESSIONID=DC7F5AB08264A51FBCDB836393CB16E7; PSESSIONID=28b334905ab6305f7a7fe051e83857bc280af1a9; __utmc=47547066; __utmb=47547066.15.10.1324693472; ACTION_RESULT_CODE=ACTION_RESULT_FAIL; ACTION_ERROR_TEXT=java.lang.NullPointerException') req.add_header('Content-Type','application/x-www-form-urlencoded') response = urllib2.urlopen(req) page = response.read() url2 = 'http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194' log_in() response2 = urllib2.urlopen(url2) page2 = response2.read() ; ACTION_ERROR_TEXT import urllib2 import urllib def log_in(): url = 'https://www.locationary.com/index.jsp?ACTION_TOKEN=tile_loginBar_jsp$JspView$LoginAction' values = {'inUserName' : ' me@gmail.com ', 'inUserPass' : 'myPass'} data = urllib.urlencode(values) req = urllib2.Request(url, data) req.add_header('Host', 'www.locationary.com') req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0') req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8') req.add_header('Accept-Language', 'en-us,en;q=0.5') req.add_header('Accept-Encoding','gzip, deflate') req.add_header('Accept-Charset','ISO-8859-1,utf-8;q=0.7,*;q=0.7') req.add_header('Connection','keep-alive') req.add_header('Referer','http://www.locationary.com/') req.add_header('Cookie','site_version=REGULAR; __utma=47547066.1079503560.1321924193.1322707232.1324693472.36; __utmz=47547066.1321924193.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); nickname=jacob501; locaCountry=1033; locaState=1795; locaCity=Montreal; jforumUserId=1; PMS=1; TurnOFfTips=true; Locacookie=enable; __utma=47547066.1079503560.1321924193.1322707232.1324693472.36; __utmz=47547066.1321924193.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); nickname=jacob501; PMS=1; __utmb=47547066.15.10.1324693472; __utmc=47547066; JSESSIONID=DC7F5AB08264A51FBCDB836393CB16E7; PSESSIONID=28b334905ab6305f7a7fe051e83857bc280af1a9; __utmc=47547066; __utmb=47547066.15.10.1324693472; ACTION_RESULT_CODE=ACTION_RESULT_FAIL; ACTION_ERROR_TEXT=java.lang.NullPointerException') req.add_header('Content-Type','application/x-www-form-urlencoded') response = urllib2.urlopen(req) page = response.read() url2 = 'http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194' log_in() response2 = urllib2.urlopen(url2) page2 = response2.read()

but it didn’t work.

EDIT 3: Tony code that doesn't work for me.

 import urllib2 import urllib import cookielib data = urllib.urlencode({"inUserName":"MYUSERNAMESHOULDBEHERE", "inUserPass":"MYPASSWORDSHOULDBEHERE"}) jar = cookielib.FileCookieJar("cookies") opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) request = urllib2.Request("https://www.locationary.com/index.jsp?ACTION_TOKEN=tile_loginBar_jsp$JspView$LoginAction", data) opener.open(request) url = "http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1012432546&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Fdennys-13470813%3Flid%3D13470813" anything = opener.open(url) anything.read()

FINAL IMAGE! I finally got it to work with Tony's suggestions!

This is my last code that worked:

 import urllib2 import urllib import cookielib data = urllib.urlencode({"inUserName":" myemail@gmail.com ", "inUserPass":"mypassword"}) jar = cookielib.FileCookieJar("cookies") opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) opener.addheaders.append(('User-agent', 'Mozilla/4.0')) opener.addheaders.append( ('Referer', 'http://www.hellboundhackers.org/index.php') ) opener.addheaders.append(('Cookie','site_version=REGULAR; __utma=47547066.912030359.1322003402.1324688192.1324930160.55; __utmz=47547066.1324655802.52.13.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=cache:dr23PN5fUj4J:www.locationary.com/%20locationary; nickname=jacob501; jforumUserId=1; PMS=1; locaCountry=1033; locaState=1786; locaCity=Vancouver; JSESSIONID=A8F241E1924CE7A25FAA8C5CA6597697; PSESSIONID=5c21c44245f978b917f17982c944a9ec2b5d2df5; Locacookie=enable; __utmb=47547066.5.10.1324930160; __utmc=47547066')) request = urllib2.Request("https://www.locationary.com/index.jsp?ACTION_TOKEN=tile_loginBar_jsp$JspView$LoginAction", data) response = opener.open(request) url = "http://www.locationary.com/" anything = opener.open(url) anything.read() ; PSESSIONID = 5c21c44245f978b917f17982c944a9ec2b5d2df5; Locacookie = enable; __utmb = 47547066.5.10.1324930160 import urllib2 import urllib import cookielib data = urllib.urlencode({"inUserName":" myemail@gmail.com ", "inUserPass":"mypassword"}) jar = cookielib.FileCookieJar("cookies") opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) opener.addheaders.append(('User-agent', 'Mozilla/4.0')) opener.addheaders.append( ('Referer', 'http://www.hellboundhackers.org/index.php') ) opener.addheaders.append(('Cookie','site_version=REGULAR; __utma=47547066.912030359.1322003402.1324688192.1324930160.55; __utmz=47547066.1324655802.52.13.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=cache:dr23PN5fUj4J:www.locationary.com/%20locationary; nickname=jacob501; jforumUserId=1; PMS=1; locaCountry=1033; locaState=1786; locaCity=Vancouver; JSESSIONID=A8F241E1924CE7A25FAA8C5CA6597697; PSESSIONID=5c21c44245f978b917f17982c944a9ec2b5d2df5; Locacookie=enable; __utmb=47547066.5.10.1324930160; __utmc=47547066')) request = urllib2.Request("https://www.locationary.com/index.jsp?ACTION_TOKEN=tile_loginBar_jsp$JspView$LoginAction", data) response = opener.open(request) url = "http://www.locationary.com/" anything = opener.open(url) anything.read()

All I had to do was add a line

 opener.addheaders.append(('Cookie','site_version=REGULAR; __utma=47547066.912030359.1322003402.1324688192.1324930160.55; __utmz=

etc .. etc (really long line of code, cookie)

Just in case, I also added the header "Referer" and "User-Agent".

Thank you, Tony!

+4
source share
2 answers

First you must write the url variable with quotes:

 url = "http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194"

And if you want to send a request without opening a browser, you can use urllib, just like you.

If you need authentication (it looks like you are doing this), you should send an authentication request, get cookies (use cookielib.FileCookieJar for this) and set them new. Then you can open pages and send requests.

Roughly you need something like:

 data=urllib.urlencode({"login":"your login or whatever, "pass":"password}) # be aware you need to change "login" and "pass" to names of fields in form you have. jar = cookielib.FileCookieJar("cookies") opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) request = urllib2.Request("url for authentication", data) opener.open(request) # now you should be authorized and able to send any request like logged in user, using opener url = "http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=1020634218&xxx_c_1_f_987=http%3A%2F%2Fwww.yellowpages.com%2Fpittsburgh-pa%2Fmip%2Ffamily-dollar-store-1349194%3Flid%3D1349194" anything = opener.open(url) anything.read()
+1
source 
 {"manifest":{"errorTimeout":0,"succeed":true,"errorCode":0,"serverVersion":"1.0","type":"locaaccess"},"saveResult":{"message":"You don't have permissions!","placeOpenedState":0,"isSucess":false}}

I get this from placing your urllib in my browser. You must first authenticate to the website, which, I believe, then run this command. I can’t give you instructions on how to enter the site, but if you go to the login page, it may have a form that you can emulate with a mail message through urllib2

0
source

Source: https://habr.com/ru/post/1388047/

More articles:


All Articles