Sparklyr write data in hdf or hive

Question

I tried using sparklyr to write data to hdfs or hive, but couldn't find a way. Is it even possible to write an R data frame in hdfs or a hive using sparklyr? Please note: my R and hadoop work on two different servers, so I need a way to write to remote hdf files from R.

Relationship Rahul

Answer 1

Writing a Spark table for a hive using Sparklyr:

iris_spark_table <- copy_to(sc, iris, overwrite = TRUE)
sdf_copy_to(sc, iris_spark_table)
DBI::dbGetQuery(sc, "create table iris_hive as SELECT * FROM iris_spark_table")

Answer 2

You can use sdf_copy_to to copy data to Spark, say tempTable. Then use DBI::dbGetQuery(sc, "INSERT INTO TABLE MyHiveTable SELECT * FROM tempTable")to insert dataframe entries into the hive table.