All geek questions in one place

Apache spark "failed to create any local directory"

I am trying to configure Apache-Spark on a small standalone cluster (1 Master Node and 8 Slave Nodes). I installed the "prebuilt" version of spark 1.1.0, built on top of Hadoop 2.4. I installed ssh between nodes without a password and exported several necessary environment variables. One of these variables (which is probably the most relevant):

export SPARK_LOCAL_DIRS=/scratch/spark/

I have a small piece of python code that I know works with Spark. I can run it locally - on my desktop, and not in the cluster - using

 $SPARK_HOME/bin/spark-submit ~/My_code.py

I copied the code to the cluster. Then I start all processes with a head node:

 $SPARK_HOME/sbin/start-all

And each of the slaves is displayed as working as an xxxxx process.

If then I try to run my code with the same command above:

 $SPARK_HOME/bin/spark-submit ~/My_code.py

I get the following error:

 14/10/27 14:19:02 ERROR util.Utils: Failed to create local root dir in /scratch/spark/. Ignoring this directory. 14/10/27 14:19:02 ERROR storage.DiskBlockManager: Failed to create any local dir.

I have permissions set to /scratch and /scratch/spark to 777. Any help is appreciated.

+5
source share
1 answer

The problem was that I did not realize that the node wizard also needed a directory from scratch. In each of my 8 work nodes, I created a local directory / scratch / spark, but did not do this on the main node. When adding a directory, the problem is fixed.

+3
source

Source: https://habr.com/ru/post/1205628/

More articles:


All Articles