So, ask if anyone knows a way to change the Spark properties (e.g. spark.executor.memory, spark.shuffle.spill.compress, etc.) at run time so that the changes can take effect between tasks / steps during time work ...
So, I know that ...
1) The documentation for Spark 2.0+ (and previous versions) states that once a Spark context has been created, it cannot be changed at run time.
2) SparkSession.conf.set, which can change some things for SQL, but I looked at more general and comprehensive configurations.
3) I could start a new context in the program with new properties, but here you need to actually configure the properties as soon as the work is already done.
Ideas ...
1) Did the execution of Exequier end so that he read the configuration file again, or simply got what was already configured at the beginning of the assignment?
2) Is there any command to force "update" properties in the spark context?
So hoping that there may be ways or other ideas (thanks in advance) ...
source
share