Use urllib2 cookie in Selenium

Question

I am trying to clean and interact with the site. Using BeautifulSoup, I can do MORE of what I want, but not all. Selenium should be able to handle this part. I can get it working with the Selenium Firefox Plugin. I just need to automate it now. My problem is that the area I need to interact with is behind the login prompt, which is handled through the OpenID provider.

Fortunately, I was able to use this bookmarklet to get the cookie set. javascript:void(document.cookie=prompt(document.cookie,document.cookie)); This allows me to log in to parse the page using BeautifulSoup.

This is done using this code:

 jar = cookielib.FileCookieJar("cookies") opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar)) opener.addheaders.append(("Cookie","__cfduid=<hex string>; __utma=59652655.1231969161.1367166137.1368651910.1368660971.15; __utmz=59652655.1367166137.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); PHPSESSID=<a session id>; __utmb=59652655.1.10.1368660971; __utmc=59652655")) page = opener.open(url).read() soup = BeautifulSoup(scrap1) ...parse stuff... 

At this point, the jar empty, and I need to make the final interaction (by clicking on the pairs of DIV elements and checking that the other DIV has been updated accordingly. However, I need the cookie specified above to fill in selenium so that I get registered accordingly.

How can I move the above cookie into something that selenium knows and recognizes?

I tried code like this

 for c in jar: driver.add_cookie({'name':c.name, 'value':c.value, 'path':'/', 'domain':c.domain}) 

But, since the jar empty, this does not work. Is there any way to place this cookie in the bank? Since I circumvented OpenId login with this cookie, I get nothing from the server.

Answer 1

I think you may be approaching this back. Instead of passing cookies to Selenium, why not log in directly with Selenium?

For instance:

 browser = webdriver.Firefox() username = 'myusername' password = 'mypassword' browser.get('http://www.mywebsite.com/') username_input = browser.find_element_by_id('username') #Using id only as an example password_input = browser.find_element_by_id('password') login_button = browser.find_element_by_id('login') username_input.send_keys(username) password_input.send_keys(password) login_button.click() 

This way you don’t have to worry about how to manually collect cookies.

Here you can capture the source of the page and pass it to BeautifulSoup:

 source = browser.page_source soup = BeautifulSoup(source) 

Hope this helps.