Copy file from remote server using SFTP directly to Azure blob storage

Question

I have a remote server with a bunch of static files (outside of Azure). I created a working role in Azure, and I want to use the working role to transfer these files using SFTP from my remote server directly to my blob memory account (without copying locally to the working role). Is there an established workflow / best practice on how to do this?

The closest I could find is the question: Copy file from URL to Azure BLOB

However, to use StartCopyFromBlob, I will need a public URL that does not fit.

Also, some of these files may be> 100mb or> 500mb, which should cause problems?

Thanks.

Answer 1

You might want to solve this problem by installing an FTP server on Azure, which saves files directly to Azure storage.

A good explanation of how to do this using preview files can be found at http://fabriccontroller.net/blog/posts/deploying-a-load-balanced-high-available-ftp-server-with-azure-files/

Answer 2

I'm not sure if downloading from SFTP to Azure Storage is possible. This feature is currently one of the best queries on the Azure Storage feedback site. (As of October 2015)

http://feedback.azure.com/forums/217298-storage

Answer 3

The easiest approach is to install the Azure command-line interface directly on the remote server. You can then use the Azure command-line interface to transfer these files directly to Azure Blob Storage. There is no need for SFTP (Azure Blob Storage does not provide the SFTP interface) or work roles (your remote server does not serve files through the web interface).

If this is not an option, another approach is to do what @Mark Volders suggested and prepare the SFTP server in Azure. Then you can send files from the remote server to the SFTP server. The SFTP server then transfers the file to the Azure Blob Storage using the Azure command line interface and deletes the local file if successful.

For an SFTP server, one obstacle is that files are copied to the Azure BLOB storage as soon as the SFTP client completes the file transfer. A common approach is to use Incron , which is a service that listens for file events (in this case, the IN_CLOSE_WRITE event). There is an SFTP Gateway product on the Azure Marketplace that does all this (disclosure: I am one of the developers of this product), so you don’t have to spend time implementing it from scratch.

In addition, file sizes> 500 MB should not be a problem for either the Azure CLI or SFTP.