Can SparkContext and StreamingContext coexist in the same program?

Question

I am trying to configure Sparkstreaming code that reads a string from a Kafka server, but processes it using rules written in another local file. I am creating streamingContext for streaming data and sparkContext for others using all other spark functions - for example, string manipulation, reading local files, etc.

val sparkConf = new SparkConf().setMaster("local[*]").setAppName("ReadLine") val ssc = new StreamingContext(sparkConf, Seconds(15)) ssc.checkpoint("checkpoint") val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2) val sentence = lines.toString val conf = new SparkConf().setAppName("Bi Gram").setMaster("local[2]") val sc = new SparkContext(conf) val stringRDD = sc.parallelize(Array(sentence)) 

But this causes the following error:

 Exception in thread "main" org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at: org.apache.spark.SparkContext.<init>(SparkContext.scala:82) org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:874) org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81) 

Answer 1

There can only be one SparkContext one SparkContext . StreamingContext is created on SparkContext . Just need to create ssc StreamingContext using SparkContext

 val sc = new SparkContext(conf) val ssc = new StreamingContext(sc, Seconds(15)) 

If the following constructor is used.

 StreamingContext(conf: SparkConf, batchDuration: Duration) 

Internally creating another SparkContext

 this(StreamingContext.createNewSparkContext(conf), null, batchDuration) 

SparkContext can get from StreamingContext from

 ssc.sparkContext 

Answer 2

yes you can do this you must first start the spark session and then use its context to start the streaming context.

val spark = SparkSession.builder (). appName ("someappname"). config ("spark.sql.warehouse.dir", storeLocation) .getOrCreate ()

val ssc = new StreamingContext (spark.sparkContext, Seconds (1))

Plain!!!