Spark Container goes beyond physical limits

Question

I searched a lot for a solution to the following problem. I use Scala 2.11.8and Spark 2.1.0.

Application application_1489191400413_3294 failed 1 times due to AM Container for appattempt_1489191400413_3294_000001 exited with exitCode: -104
For more detailed output, check application tracking page:http://ip-172-31-17-35.us-west-2.compute.internal:8088/cluster/app/application_1489191400413_3294Then, click on links to logs of each attempt.
Diagnostics: Container [pid=23372,containerID=container_1489191400413_3294_01_000001] is running beyond physical memory limits. 
Current usage: 1.4 GB of 1.4 GB physical memory used; 3.5 GB of 6.9 GB virtual memory used. Killing container.

Please note that in this error I indicated a lot more than 1.4 GB. Since I see that none of my artists failed, I read this error: the driver needs more memory. However, my settings do not seem to apply.

I set the parameters for the yarn as follows:

val conf = new SparkConf()
  .setAppName(jobName)
  .set("spark.hadoop.mapred.output.committer.class", "com.company.path.DirectOutputCommitter")
additionalSparkConfSettings.foreach { case (key, value) => conf.set(key, value) }

// this is the implicit that we pass around
implicit val sparkSession = SparkSession
  .builder()
  .appName(jobName)
  .config(conf)
  .getOrCreate()

where the memory provisioning options additionalSparkConfSettingswere set with the following snippet:

HashMap[String, String](
  "spark.driver.memory" -> "8g",
  "spark.executor.memory" -> "8g",
  "spark.executor.cores" -> "5",
  "spark.driver.cores" -> "2",
  "spark.yarn.maxAppAttempts" -> "1",
  "spark.yarn.driver.memoryOverhead" -> "8192",
  "spark.yarn.executor.memoryOverhead" -> "2048"
)

Are my settings really not valid? Or am I misinterpreting magazines?

Thank!

Answer 1

, .

. , -, , . , .

, , . , .

Answer 2

, , .

spark.yarn.executor.memoryOverhead = executorMemory * 0.10, with minimum of 384 

( ), . , , , , .. ( 6-10%).

spark.yarn.driver.memoryOverhead = driverMemory * 0.10, with minimum of 384.

( ), . , , VM, , .. ( 6-10%).

, .

. SO

!