Spark with HBASE vs. Spark with HDFS

Question

I know that HBASE is a columnar database that stores structured table data in HDFS by column rather than row. I know that Spark can read / write from HDFS and that there is some HBASE connector for Spark, which can now also read and write HBASE tables.

Questions:

1) What are the additional features associated with splitting Spark on top of HBASE instead of using HBASE? Does it depend only on the capabilities of the programmer or are there any improvements in performance? Are there any things that can be fixed, and HBASE cannot exclusively do?

2) Based on the previous question, when should you add HBASE between HDFS and SPARK instead of direct HDFS?

Answer 1

1) What are the added features caused by Spark stratification over HBASE instead of using only HBASE? Does it depend only on the programmer's capabilities or are there any reasons for this? Are there things that can spark, and HBASE cannot do exclusively?

At Splice Machine, we use Spark for our analytics on top of HBase. HBase does not have a execution mechanism, and the spark provides a competent mechanism for executing on top of HBase (intermediate results, relational algebra, etc.). HBase is the MVCC storage structure, and Spark is the execution engine. They are natural additions to each other.

2) Based on the previous question, when should you add HBASE between HDFS and SPARK instead of direct HDFS?

, /, ( )

...

Answer 2

Hbase - No SQL, , . , Hfile ( HDFS) .

, Hbase, , .

Spark, , , hdfs, hbase, hive, postgreSQL, json , .. HDFS Hbase gbs . Hbase ....

Answer 3

, , Apache Hadoop Apache Spark, . .

HBase , HDFS, , .

HBase Hadoop Spark, , - ! HFiles, .

, SQL, , . , (). NoSQL - , , (, , NoSQL) - . , SSD , - . , .

:

• .

• I think that if you use Apache Spark for data analysis, you need to avoid HBase (Cassandra or any other database). They can be useful for storing aggregated data for reporting or selecting specific records about users or items, but this happens after processing.