Use powershell to automate downloading files from a website (don't know the URL for the file)

Question

I want to automate the daily download of a report from our credit card provider website.

The file is dynamically generated, so I do not know the actual URL of the file itself.

I can use powershell to navigate and enter a web page, and also click the "Download report" button. But then the File Download dialog box appears, and I can’t figure out how to click the Save button in this dialog box.

It looks like my options are:

• Somehow find this window and try sending keystrokes

• Somehow find the file url when a dialog box appears, then upload directly.

• Find a way to disable the File Download field, i.e. (putting it on trusted sites didn't work)

I use:

$ie = New-Object -com "InternetExplorer.Application" 

What would you suggest is the best approach?

Answer 1

For this purpose, the best option is to ask the provider (if possible) to publish some API.

For this purpose, sending keystrokes is very fragile. What after clicking the "Save" button? A dialog box will open, and you should go to the folder and click "OK." This means you need to find the window again. But what if some kind of similar window opens? Will your script find the right window? (for other scenarios where you need to hide / show / find windows and send keystrokes, I would recommend Autohotkey .)

Other features depend on which site it is. It seems that the file is being generated as a response to the POST command. This means that there is no direct link that you could download. And even if there was a link, it is likely that it could not be downloaded via Net.WebClient , because you would not send any authentication information (session cookies or session identifier in the query string). You will need to make out, for example. cookies from $ie.Document.cookie and add them to the request.

However, in some cases, you can use [System.Net.WebRequest]$webRequest = [System.Net.WebRequest]::Create($url); and specify the POST method and data and wait for a response (which will be a file). This would emulate a click on the submit button. But then again, without credentials, this is likely to fail.

Without access to the site, it’s hard to say anything else.

Answer 2

If you are absolutely NEED to programmatically interact with the application, I would suggest using something like http://www.codeplex.com/white

This library provides a wrapper for some of the automation features provided by .NET 3.5.

As stated above, the best approach is to use Net.WebClient instead of an instance of IE. If you need to restore a GET or POST request, you can use Firebug to check the page provided by your provider.