Spark broadcasting cassandra connector

Question

I am using spark-cassandra-connector 1.1.0 provided by datastax. I noticed an intermediate problem, and I'm not sure if something like this happens: When I translate the cassandra connector and try to use it for artists, I get an exception, assuming that my configuration is invalid, cannot connect to Cassandra at 0.0.0.

Sample stack:

java.io.IOException: Failed to open native connection to Cassandra at {0.0.0.0}:9042
        at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:174)
        at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:160)
        at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:160)
        at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:36)
        at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:61)
        at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:71)
        at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:97)
...

But if I use it without translation, everything works fine.

Which is also strange for me, on the side of the transmitter the value of the correct configuration is transmitted, but on the side of the executor there is no.

Driver side:

  val dbConf = ssc.sparkContext.getConf
  val connector = CassandraConnector(dbConf)
  println(connector.hosts) //Set(10.20.1.5) 
  val broadcastedConnector = ssc.sparkContext.broadcast(connector)
  println(broadcastedConnector.value.hosts) //Set(10.20.1.5) 

Implementing Party:

mapPartition{
...
 println(broadcastedConnector.hosts) // Set(0.0.0.)
...
}

Can anyone explain why it works this way and how to translate the Cassandra connector in a way that can be used on the performers side.

. 1.2.3.

Answer 1

Cassandra Connector. , .