Download file from Internet via R despite popup

Question

Downloading a file from the Internet using R is simple and has been previously reviewed .

My question is about how to get through a pop-up message that does not seem to allow the download to complete. In particular,

download.file(url = "https://www.chicagofed.org/applications/bhc_data/bhcdata_index.cfm?DYR=2012&DQIR=4", destfile = "data/test.zip")

gives me a small garbage file instead of the desired 18 megabyte file that you will get if you go to the website and enter the year 2012and quarter 4manually. I suspect the problem is that, as you can see, when you do this manually, a pop-up window interrupts the download process, asking if you want to save the file or open it. Is there a way to automatically pop-up a window (i.e., through download.file)?

Answer 1

This can be done using Selenium, see https://github.com/ropensci/RSelenium .

require(wdman)
require(RSelenium)


selPort <- 4444L
fprof <- makeFirefoxProfile(list(browser.download.dir = "C:\\temp"
                                 ,  browser.download.folderList = 2L
                                 , browser.download.manager.showWhenStarting = FALSE
                                 , browser.helperApps.neverAsk.saveToDisk = "application/zip"))
selServ <- selenium(port = selPort)
remDr <- remoteDriver(extraCapabilities = fprof, port = selPort)
remDr$open(silent = TRUE)
remDr$navigate("https://www.chicagofed.org/applications/bhc_data/bhcdata_index.cfm")
# click year 2012
webElem <- remDr$findElement("name", "SelectedYear")
webElems <- webElem$findChildElements("css selector", "option")
webElems[[which(sapply(webElems, function(x){x$getElementText()}) == "2012" )]]$clickElement()

# click required quarter

webElem <- remDr$findElement("name", "SelectedQuarter")
Sys.sleep(1)
webElems <- webElem$findChildElements("css selector", "option")
webElems[[which(sapply(webElems, function(x){x$getElementText()}) == "4th Quarter" )]]$clickElement()

# click button

webElem <- remDr$findElement("id", "downloadDataFile")
webElem$clickElement()

Answer 2

Please install the firebug add-on in your firefox and find out what happens when you vistit and configure the web request. IMO is a request for 2013, which is more complex and needs a detailed analysis. It uses cookies and runs some script actions ...

[23: 26: 52.593] GET https://ews-sdc.federalreserve.org/dcslfh67p000004nku46ap9ku_7w8e/dcs.gif?&dcsdat=1393021612511&dcssip=www.chicagofed.org&dcsuri=https://www.chicagofed.org/applications/bhc_data/bhcdata_index.cfm&dcsref=https://www.chicagofed.org/applications/bhc_data/bhcdata_index.cfm%3FDYR=2012%26DQIR=4&WT.tz=1&WT.bh=23&WT.ul=de-DE&WT.cd=24&WT.sr=1920x1080&WT.jo=Yes&WT.ti=FormButton:BHCDATA&WT.js=Yes&WT.jv=1.8&WT.ct=unknown&WT.bs=1920x570&WT.fv=11.2&WT.slv=Not%20enabled&WT.tv=9.4.0&WT.dl=27&WT.ssl=1&WT.es=www.chicagofed.org/applications/bhc_data/bhcdata_index.cfm&WT.ce=2&WT.vt_f_tlh=1393021608&WT.vtvs=1393019653663&WT.vtid=84.139.146.195-3993071712.30355278&WT.co_f=84.139.146.195-3993071712.30355278&WT.nv=content [HTTP/1.1 200 OK 202ms]

Answer 3

Here is an update of @jdharrison's answer, as it startServerno longer works.

require(wdman)
require(RSelenium)

selPort <- 4444L

fprof <- makeFirefoxProfile(list(browser.download.dir = "C:\\temp"
                                 ,  browser.download.folderList = 2L
                                 , browser.download.manager.showWhenStarting = FALSE
                                 , browser.helperApps.neverAsk.saveToDisk = "application/zip"))

selServ <- selenium(port = selPort)

remDr <- remoteDriver(extraCapabilities = fprof, port = selPort)

Then continue as in the original answer.