All geek questions in one place

Spark working memory

I installed a standalone spark cluster (1.6). have 1 master and added 3 machines in the conf / slaves file as workers. Despite the fact that I allocated 4 GB of memory to each of my employees in sparks, why does it use only 1024 MB when launching the application? I would like him to use all 4 GB allocated to him. Help me figure out where and what I'm doing wrong.

Below is a screenshot of the spark's main page (when the application is running using spark-submit), where in the Memory column it displays 1024.0 MB, which is used in brackets next to 4.0 GB.

I also tried to install the option - executor-memory 4G with the spark-submit function, and it does not work (as suggested in How to change memory on node for apache spark worker ).

These are the parameters that I set in spark-env.sh

export SPARK_WORKER_CORES = 3

export SPARK_WORKER_MEMORY = 4g

export SPARK_WORKER_INSTANCES = 2

enter image description here

+5
source share
3 answers

Another solution is to try to set the following parameters inside the conf/spark-defaults.conf file:

 spark.driver.cores 4 spark.driver.memory 2g spark.executor.memory 4g

After you installed above (only the last line in your case), immediately disable all workers and restart them. It is better to initialize the memory of the performers in this way, since your problem is that not a single performer can allocate all the available memory of his worker.

+3
source

The parameter you are looking for, executor-memory try to turn on this spark application when it starts.

--executor-memory 4g

When you set the working memory to 4g, then the largest executor that you run for this working memory is 4g. PS: you can have different configurations (each of which has a different working memory).

0
source

create a file called spark-env.sh in the spark / conf directory add this line SPARK_EXECUTOR_MEMORY = 4g

0
source

Source: https://habr.com/ru/post/1246426/

More articles:


All Articles