Configure Apache Spark logging with Scala and logback

Question

I am very confused about setting up logging with Apache Spark. Outcome Apache used Log4j for logging and generates a huge amount of log data. Is there a way to configure log4j for spark logs and use logback for application log. I am pretty familiar with logback, but it seems the spark only supports log4j. Below, the code snippet worked fine until I introduced the apache spark. Any help in this regard would be helpful.

import com.typesafe.scalalogging.LazyLogging

import scala.util.{Failure, Success}
import scala.xml.{Elem, XML}



object MainApp extends App with LazyLogging  {



  val currency = new YahooCurrencyLoader() with CurrencyParameters
  val ccy = currency.getXML(currency.ccyUrl) match {

    case Success(v) => XML.save("PreviousRun.xml",v); logger.info("XML has been saved for use")
    case Failure(ex) => logger.error("XML extraction failed. Look at Yahoo extraction class. ${ex.getMessage}" )

  }

  val xmllocation: String = "./PreviousRun.xml"
  val loadxml: Elem = XML.loadFile(xmllocation)
  //print(loadxml)
  //print(currency.findCurrency(loadxml,"GBP"))

  logger.info("USD CAD Cross is " + currency.findCurrency(loadxml,"CAD").head)

Answer 1

, sbt maven, . sbt, , .

1. Apache Spark log4j 1.2.xx

, , . !

-, libs spark:

• log4j
• slf4j-log4j12

sbt ( sbt-assembly) :

lazy val spark16 = Seq("spark-core", "spark-sql", "spark-hive")
  .map("org.apache.spark" %% _ % "1.6.1")
  .map(_.excludeAll(
    ExclusionRule(name = "log4j"),
    ExclusionRule(name = "slf4j-log4j12")
  ))

2. log4j slf4j

: https://www.slf4j.org/legacy.html
, : log4j-over-slf4j

log4j-over-slf4j log4j, org.apache.log4j.Category, org.apache.log4j.Logger, org.apache.log4j.Priority, org.apache.log4j.Level, org.apache.log4j.MDC org.apache.log4j.BasicConfigurator. SLF4J.

, slf4j, .

"org.slf4j" % "log4j-over-slf4j" % "1.7.25"

3.

( ) logback, :

"ch.qos.logback" % "logback-classic" % "1.2.3"

logback.xml , , src/main/resources !

logback spark-submit, : fooobar.com/questions/1669207/...

Answer 2

. import org.slf4j.Logger

import org.slf4j.LoggerFactory

, . Object SparkCode{

val logger = LoggerFactory.getLogger(this.getClass.getName)

def main(args: Array[String]) {

logger.info("Connection Started . ")

// Create Spark Context and go on..

}

}

.