How to automate the loading of satellite images?

Question

I am looking for a way to automate the process of loading satellite imagery. The screenshot shows the type and format of the files that interest me when downloading (.ntf and 150 MB files).

I came across the following code from TheBioBucket , which looks promising, although the R XML package is deprecated.

require(XML) dir.create("D:/GIS_DataBase/DEM/") setwd("D:/GIS_DataBase/DEM/") doc <- htmlParse("http://www.viewfinderpanoramas.org/dem3.html#alps") urls <- paste0("http://www.viewfinderpanoramas.org", xpathSApply(doc,'//*/a[contains(@href,"/dem1/N4")]/@href')) names <- gsub(".*dem1/(\\w+\\.zip)", "\\1", urls) for (i in 1:length(urls)) download.file(urls[i], names[i]) 

Is there a good way to automate the process of loading .ntf files programmatically using R or Python?

Answer 1

Scraper is definitely easy to implement in Python.

 # collect.py import urllib, urllib2, bs4 from urlparse import urljoin soup = bs4.BeautifulSoup(urllib2.urlopen("http://www.viewfinderpanoramas.org/dem3.html#alps")) links = soup.find_all('a') for link in links: try: if "/dem1/N4" in link['href']: url = urljoin("http://www.viewfinderpanoramas.org/", link['href']) filename = link['href'].split('/')[-1] urllib.urlretrieve(url, filename) #break except: pass 

You might want to change the file name to include the path where you want to put the file

Answer 2

In R an XML package can facilitate what you need quite easily. Here is a place to start

 library(XML) demdir <- "http://www.viewfinderpanoramas.org/dem1/" # this returns a data.frame with file names dems <- readHTMLTable(demdir)[[1]] # you'll want, for example, to download only zip files demnames <- dems[grepl(".zip",dems$Name),"Name"] # (but you can add other subsetting/selection operations here) # download the files to the same name locally # (change '.' if you want some other directory) sapply(demnames, function(demfi) download.file(paste0(demdir,demfi), file.path(".",demfi))) 

The only complication I see is if the file name is too long (if it is truncated in your web browser), then the file name in dems will also be truncated.