Skip system property to fix-send and read file from class path or custom path

Question

I recently found a way to use logback instead of log4j in Apache Spark (both for local use and for spark-submit). However, the last fragment is missing.

The problem is that Spark really does not see the settings logback.xmlin its path to the classes. I already found a way to load it at local runtime:

What am i still

Basically, checking the System property logback.configurationFile, but loading logback.xmlfrom mine /src/main/resources/just in case:

// the same as default: https://logback.qos.ch/manual/configuration.html
private val LogbackLocation = Option(System.getProperty("logback.configurationFile"))
// add some default logback.xml to your /src/main/resources
private lazy val defaultLogbackConf = getClass.getResource("/logback.xml").getPath

private def getLogbackConfigPath = {
   val path = LogbackLocation.map(new File(_).getPath).getOrElse(defaultLogbackConf)
   logger.info(s"Loading logging configuration from: $path")
   path
}

And then when I initialize my SparkContext ...

val sc = SparkContext.getOrCreate(conf)
sc.addFile(getLogbackConfigPath)

I can confirm that it works locally.

Play with spark-submit

spark-submit \
  ...
  --master yarn \
  --class com.company.Main\
  /path/to/my/application-fat.jar \
  param1 param2 

This gives an error:

Exception in thread "main" java.io.FileNotFoundException: Added file file:/path/to/my/application-fat.jar!/logback.xml does not exist

, , ( )

getClass.getResource("/logback.xml").getPath

,

sc.addFile(getLogbackConfigPath)

... ! !? !? . , , .

spark-submit

, , . , . logback.xml application-fat.jar :

spark-submit \
  ...
  --conf spark.driver.extraJavaOptions="-Dlogback.configurationFile=/path/to/my/logback.xml" \
  --conf spark.executor.extraJavaOptions="-Dlogback.configurationFile=/path/to/my/logback.xml" \
  --master yarn \
  --class com.company.Main\
  /path/to/my/application-fat.jar \
  param1 param2 

, . ! ?

-Dlogback.configurationFile

?

!

Answer 1

1. java.io.FileNotFoundException

.

SparkContext.addFile Jar. , zip .

Fine.

2. -Dlogback.configurationFile

- .

--master yarn, --deploy-mode cluster client.

https://spark.apache.org/docs/1.6.1/configuration.html#application-properties

spark.driver.extraJavaOptions

. SparkConf , JVM . -driver-java-options .

, --driver-java-options:

spark-submit \
  ...
  --driver-java-options "-Dlogback.configurationFile=/path/to/my/logback.xml" \
  --master yarn \
  --class com.company.Main\
  /path/to/my/application-fat.jar \
  param1 param2 

--driver-java-options

--conf , :

--driver-java-options "-Dlogback.configurationFile=/path/to/my/logback.xml -Dother.setting=value" \

--driver-java-options "-Dlogback.configurationFile=/path/to/my/logback.xml" \
--driver-java-options "-Dother.setting=value" \