How to change SparkContext.sparkUser () parameter (in pyspark)?

Question

I am new to Sparkand pyspark.
I use pyspark after my processing rdd, I tried to save it in hdfsusing a function saveAsTextfile(). But I get the resolved error message "because pyspark is trying to write hdfs using my local account" kjlee ", which does not exist on the system hdfs.

I can check the spark username for SparkContext().sparkUser(), but I cannot find how to change the spark username.

How can I change spark username?

Answer 1

: HADOOP_USER_NAME HADOOP_USER_NAME = anyuser pyspark os.environ [ "HADOOP_USER_NAME" ] = "anyuser"

Answer 2

Scala System.setProperty:

  System.setProperty("HADOOP_USER_NAME","newUserName")

  val spark = SparkSession
    .builder()
    .appName("SparkSessionApp")
    .master("local[*]")
    .getOrCreate()

  println(spark.sparkContext.sparkUser)