Automate login and form filling?

Question

I am trying to enter the site and automatically save the HTML page (I want this to be possible at a regular time interval). From the surface, this is a typical modern website where, if a user goes directly to a "blocked" URL, a login form appears, and after logging in, the user is redirected to the intended page.

I gave a mechanized snapshot ( http://wwwsearch.sourceforge.net/mechanize/ ), but he didn’t find some form elements needed to enter (hidden elements that have some values ​​entered by the javascript function, which runs when the user clicks the " to come in" ).

I played a little with the "web browser" in .NET, but quickly lost interest, because I could not even get him to send a request to a Google page.

I don't care what language is; I will learn to solve this problem. At a minimum, it should work on Windows.

A simple example, for example, entering text in a Google search box will be a great bonus.

Answer 1

In my experience, the most reliable way is to use javascript. It works well in .Net. To check, go to the following addresses one after another in Firefox or Internet Explorer:

http://www.google.com javascript:function f(){document.forms[0]['q'].value='stackoverflow';}f(); javascript:document.forms[0].submit() 

Searches for "stackoverflow" on Google. To do this in VB.Net using the webbrowser control, do the following:

 WebBrowser1.Navigate("http://www.google.com") Do While WebBrowser1.IsBusy OrElse WebBrowser1.ReadyState <> WebBrowserReadyState.Complete Threading.Thread.Sleep(1000) Application.DoEvents() Loop WebBrowser1.Navigate("javascript:function%20f(){document.forms[0]['q'].value='stackoverflow';}f();") Threading.Thread.Sleep(2000) 'wait for javascript to run WebBrowser1.Navigate("javascript:document.forms[0].submit()") Threading.Thread.Sleep(2000) 'wait for javascript to run 

Notice how the space in the URL is converted to% 20. I am not sure if this is necessary, but it cannot hurt. It is important that the first javascript is in the function. Sleep () calls must wait for Google to load, as well as for javascript. Do While Loop can run forever if the page does not load, so for automation purposes there is a counter that will time out, say, after 60 seconds.

Of course, for Google you can simply go directly to www.google.com?q=stackoverflow, but if your site has hidden input fields, etc., then this is the way to go. Only works for HTML sites - flash is a completely different matter.

Answer 2

If you understand correctly, you want to log in to only one web page, and this form always remains the same. You can either reverse engineer the java script or debug it using the javascript debugger in the browser (e.g. firebug for firefox). Or you can fill out the form in your browser and watch the http request using a network analyzer. Once you have all the necessary form data to submit, you can do the same with your program (this is what I did the last time I had a pretty similar task). Remember to save all the cookie data that you requested back from the web server and send it with the following request in order to “stay logged in”.

Answer 3

It has already been discussed here .

Basically, its essence is that you can use selenium , an open source web automation tool that has an api library available in different languages, for example java, ruby, etc.

Answer 4

Neoload can handle filling out a form with authentication, assuming you don't want to collect data, just follow the steps. This is a tool for web stress, so it is not intended to be used as a time-based service, but you CAN leave it operational.

Answer 5

I used Ruby and Watir (a web application testing suite) for something similar, but it was a very small task (basically visiting the URLs from a text file and loading the image).

There's also an iMacros extension that can do some automation, but I personally am not familiar with it (I just know).

Answer 6

"I'm trying to log in to the site and automatically save the HTML page"

  SAVEAS TYPE=HTM FOLDER=C: FILE=page.html 

https://addons.mozilla.org/en-US/firefox/addon/imacros-for-firefox/?src=search

These commands, executed in the iMacros addon, will save the page to C: drive and name it page.html

Moreover,

 URL GOTO=www.website.com 

Going to the specific website you want to save. You can also use scripts in iMacros and set up various websites in a macro.