Spark - java IOException: Failed to create local directory in / tmp / blockmgr *

Question

I tried to run a long Spark Job. After several hours of execution, I get the exception below:

Caused by: java.io.IOException: Failed to create local dir in /tmp/blockmgr-bb765fd4-361f-4ee4-a6ef-adc547d8d838/28 

Tried to get around this by checking:

• Resolving the problem in the / tmp directory. The source server does not work as root. but / tmp dir should be writable by all users.

• / tmp Dir has enough space.

Answer 1

Assuming that you are working with several nodes, you will need to check each node involved in the spark operation (master / driver + subordinates / nodes / workers).

Please confirm that each working / node has enough disk space (especially in the check / tmp folder) and in the right resolution.

Answer 2

Back up the tmp cache file. Clean the tmp directory and run your shell. This will work. The same approach worked for me.

Answer 3

Edit: The answer below does not ultimately solve my case. This is because some Spark subfolders (or some of its dependencies) were created, but not all of them. The frequent need to create such paths will make any project unviable. So I launched Spark (PySpark in my case) as an administrator, who solved the problem. So, in the end, this is probably a permissions issue.

---

Original answer:
I solved the same problem as on my local Windows machine (not in the cluster). Since there were no problems with permissions, I created a directory that Spark could not create, i.e. I created the following folder as a local user, and I did not need to change any permissions for this folder.

 C:\Users\<username>\AppData\Local\Temp\blockmgr-97439a5f-45b0-4257-a773-2b7650d17142 

Answer 4

After checking all permissions and user access.

I had the same problem when building components in Talend studio, and it was solved by providing the correct "/" character in the spark scratch directory (temporary directory) on the Configuration Configuration tab. This is necessary when building jar in windows and running in a Linux cluster.