Upload file weekly from FTP to HDFS

Question

I want to automate the weekly download of a file from an ftp server to a CDH5 cluster. What would be the best way to do this?

I thought about the work of the Oozie coordinator, but I can't think of a good way to upload a file.

Answer 1

Since you are using CDH5, it is worth noting that the NFSv3 interface to HDFS is included in this Hadoop distribution. You should check the " NFSv3 Gateway Configuration" in the CDH5 installation documentation.

wget, curl, python .., mount NFS. , ... "" "". , (python script, curl, ftp ..) ${myVar}.

, .

Answer 2

, .

, , .

http://oozie.apache.org/docs/3.3.0/DG_ShellActionExtension.html

script,

wget http://myftp.com/file.name

, , script