Good practice for providing only one Spark context per application

Question

I am looking for a good way to ensure that my application uses only one Spark Context (sc). During development, I often encounter errors and must restart my game! server to check my changes. Would a Singleton template be a solution?

object sparckContextSingleton {
  @transient private var instance: SparkContext = _
  private val conf : SparkConf = new SparkConf()
    .setMaster("local[2]")
    .setAppName("myApp")

  def getInstance(): SparkContext = {
    if (instance == null){
      instance = new SparkContext(conf)
    }
    instance
  }  
}

It does not do a good job. Should I stop SparkContext?

Answer 1

That should be enough to do the trick; it's important to use val, not var.

object SparkContextKeeper {
  val conf = new SparkConf().setAppName("SparkApp")
  val context= new SparkContext(conf)
  val sqlContext = new SQLContext(context)
}

Answer 2

On Play, you should write a plugin that issues SparkContext. Use the start and stop buttons of the plugin to start and stop the context.