Question on using C # for discussion with Hadoop FileSystem

Question

Currently, my application uses C # with MONO on Linux to communicate with local file systems (e.g. ext2, ext3). Basic operations: open the file, write / read from the file, and close / delete the file. To do this, I am currently using native C # APIs (e.g. File.Open) to work with the file.

My question is if I install the Hadoop file system on my Linux box. then what change do I need to make with my existing functions so that they exchange the hadoop file system to do basic file operations. Since the Hadoop infrastructure is based on Java, like any C # application (with MONO on linux) will perform basic operations with Hadoop. Do the main APIs in C # use a file (likr File.Open or File.Copy) to work well with Hadoop file systems?

I thought something like this: Since Hadoop provides a C API for file operations. So write a C wrapper and insert the DLL from it. Then use this DLL in C # code to communicate with the Hadoop file systems.

Is it correct? or Someone may suggest some kind of document or steps so that my C # programs can open / read / write files from Hadoop FileSystems.

Thanks, Anil.

Answer 1

Hadoop supports installing HDFS through a fuse: http://wiki.apache.org/hadoop/MountableHDFS This is probably a simpler solution than packing native C libraries, although this approach also works.

Answer 2

You can also use the Thrift interface for HDFS to get the C # client. See http://wiki.apache.org/hadoop/HDFS-APIs for more details .

FUSE, , , HDFS .

, WebDAV, HDFS Windows. wiki:

CDH2 + WebDav.

• HDFS-over-Webdav

   1. git clone git://github.com/huyphan/HDFS-over-Webdav.git
   2. Set HDFS_WEBDAV_SRC_DIR to the path you cloned it to
  

• conf/hadoop-webdav.sh

  export HADOOP_WEBDAV_HOST = xxx.xxx.xxx.xx # /IP NN export HADOOP_WEBDAV_PORT = 9001 # export HADOOP_WEBDAV_HDFS = hdfs://localhost: 9000/# fs.default.name export HADOOP_WEBDAV_CLASSPATH = $HDFS_WEBDAV_SRC_DIR/lib # .

• Build/

  HADOOP_HOME = XXX cd HDFS-over-Webdav ant -Dhadoop.dir = $HADOOP_HOME cp bin/* $HADOOP_HOME/bin cp conf/* $HADOOP_HOME/conf

• WebDav

  cd $HADOOP_HOME. /bin/start -webdav.sh # $HADOOP_HOME/logs

•  1. Can use user@ authority syntax below if you have HDFS permissions setup
   2. XP: Add network place http://$HADOOP_WEBDAV_HOST:$HADOOP_WEBDAV_PORT/ under "My Network Places"
   3. Vista/Win7: "Map Netowork Drive" using the above ID
   4. Linux CLI: cadaver $HADOOP_WEBDAV_HOST:$HADOOP_WEBDAV_PORT
   5. Linux Nautilus: Go -> Location, use above ID
  

,