GC spark operator takes long

Question

I'm doing Spark work in an autonomous cluster, and I noticed that once the GC starts to last, and a scary red color begins to appear.

Here are the available resources:

Cores in use: 80 Total, 76 Used
Memory in use: 312.8 GB Total, 292.0 GB Used

Work Details:

spark-submit   --class com.mavencode.spark.MonthlyReports   
--master spark://192.168.12.14:7077   
--deploy-mode cluster   --supervise   
--executor-memory 16G --executor-cores 4 
--num-executors 18  --driver-cores 8 
--driver-memory 20G montly-reports-assembly-1.0.jar

How to fix GC time for so long?

Answer 1

I had the same problem and was able to solve it using Parallel GC instead of G1GC. You can add the following options for executing additional Java parameters in a submit request

-XX:+UseParallelGC -XX:+UseParallelOldGC