Spark presents yarn as another user

Question

Is it possible to send a spark task to a cluster of yarn and select, either using the command line or inside a jar, which user will “own” the task?

The spark-submit will be run from the script containing the user.

PS: is it possible if the cluster has kerberos configuration (and script a keytab)?

Answer 1

For non-kerberized cluster : export HADOOP_USER_NAME=zorrobefore submitting jobs, Spark will do the trick.
Then make sure that unset HADOOP_USER_NAMEif you want to return to the default credentials in the rest of the shell script (or in an interactive shell session).

For a clustered cluster, the clean way to impersonate another account without parsing your other tasks / sessions (which probably depends on your default ticket) will be something on this line ...

export KRB5CCNAME=FILE:/tmp/krb5cc_$(id -u)_temp_$$
kinit -kt ~/.protectedDir/zorro.keytab [e-mail]
spark-submit ...........
kdestroy

Answer 2

If your user exists, you can still run your spark submit su $ my_user -c spark submit [...]

I'm not sure about the kerberos key, but if you do kinit with this user, everything should be fine.

su, , stackoverflow: script

Answer 3

For a non-kerberization cluster, you can add Spark conf as:

--conf spark.yarn.appMasterEnv.HADOOP_USER_NAME=<user_name>