All geek questions in one place

Can Apache YARN be used without HDFS?

I want to use Apache YARN as a cluster and resource manager for working with an infrastructure in which resources will be shared in different tasks of the same structure. I want to use my own distributed file system with a bunch.

  • Can I use any other distributed file system with YARN except HDFS?

  • If so, which HDFS APIs need to be implemented?

  • What Hadoop components are required to run YARN?
+4
source share
5 answers

There are several different questions here.

YARN , - S3 ?

: LinkedIn Samza , http://downloads. Samza , hdf, :// , .

, , .

YARN ?

.

" ", . filesytem: (), , . () , - O (1). , ,... , HBase append().

MapR , Redhat GlusterFS; IBM EMC . , HDFS; , FS ( - , , Hortonworks Cloudera).

YARN, FS.

, FS , - . HBase - .

  • Microsoft Azure Storage , FS () s . Azure HDFS.
  • Google 1 2017 , GCS . , ; .
  • Amazon EMR s3 , (a) () , HBase .
  • S3- ASF, S3a, . , , ; s3guard, dynamo s3guard, ( , ()).

HDFS?

, !

, API. Apache Bigtop, . HBase Accumulo, : Mapreduce, Hive, spark, Flink.

Hadoop common-dev bigtop .

+7

, , , . . , Hadoop , , S3/AzureBlobs/FTP, .

, fs.defaultFS .

+2

, , API HDFS.

. AWS S3 (s3n:// s3a://) HDFS. , API HDFS.

0

- . Apache Mesos - ( .). hadoop. , dc/os ( , ..)

-1

YARN HDFS. HDFS, HDFS.

YARN Hadoop. Hadoop YARN ( , ).

-1

Source: https://habr.com/ru/post/1671223/

More articles:


All Articles