Spark: saveAsTextFile () creates only the SUCCESS file and there is no part file when writing to the local file system

I am writing RDD to a file using the following command:

rdd.coalesce(1).saveAsTextFile(FilePath) 

When FilePath is the HDFS path ( hdfs://node:9000/folder/ ), everything works fine.

When FilePath is a local path ( file:///home/user/folder/ ), everything works. An output folder is created and the SUCCESS file is SUCCESS .

However, I do not see the part-00000 file containing the output. There is no other file. There are also no errors at the output of the spark console.

I also tried calling collection on RDD before calling saveAsTextFile() , giving 777 permission to display the folder, but nothing works.

Please, help.

+5
source share

Source: https://habr.com/ru/post/1268827/


All Articles