Insert data into a Cassandra table Using a Spark DataFrame

Question

I am using Scala Version 2.10.5 Cassandra 3.0 and Spark 1.6. I want to insert data into cassandra, so I tried the Original example

scala> val collection = sc.parallelize(Seq(("cat", 30), ("fox", 40))) scala> collection.saveToCassandra("test", "words", SomeColumns("word", "count")) 

What works and the ability to insert data into Cassandra.So I had a csv file that I wanted to insert into the Cassandra table, matching the schema

 val person = sc.textFile("hdfs://localhost:9000/user/hduser/person") import org.apache.spark.sql._ val schema = StructType(Array(StructField("firstName",StringType,true),StructField("lastName",StringType,true),StructField("age",IntegerType,true))) val rowRDD = person.map(_.split(",")).map(p => org.apache.spark.sql.Row(p(0),p(1),p(2).toInt)) val personSchemaRDD = sqlContext.applySchema(rowRDD, schema) personSchemaRDD.saveToCassandra 

When I use the SaveToCassndra Iam, getting saveToCassandra is not part of personSchemaRDD. So taught in different ways.

  df.write.format("org.apache.spark.sql.cassandra").options(Map( "table" -> "words_copy", "keyspace" -> "test")).save() 

But receiving cannot connect to cassandra on ip: port.can someone will tell me the best way to do this. I need to periodically save data in cassandra from files.

Answer 1

sqlContext.applySchema(...) returns a DataFrame , and a DataFrame does not have a saveToCassandra method.

You can use the .write method:

 val personDF = sqlContext.applySchema(rowRDD, schema) personDF.write.format("org.apache.spark.sql.cassandra").options(Map( "table" -> "words_copy", "keyspace" -> "test")).save() 

If we want to use the saveToCassandra method, the best way is to have RDD with schema support using the case class.

 case class Person(firstname:String, lastName:String, age:Int) val rowRDD = person.map(_.split(",")).map(p => Person(p(0),p(1),p(2).toInt) rowRDD.saveToCassandra(keyspace, table) 

The Dataframe write method should work. Make sure you set your context correctly.