Value for HADOOP_CONF_DIR from the cluster

Question

I installed a cluster (YARN) using Ambari with 3 virtual machines as hosts.

Where can I find the value for HADOOP_CONF_DIR?

# Run on a YARN cluster
export HADOOP_CONF_DIR=XXX
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn-cluster \  # can also be `yarn-client` for client mode
  --executor-memory 20G \
  --num-executors 50 \
  /path/to/examples.jar \
  1000

Answer 1

Install Hadoop. In my case, I installed it in / usr / local / hadoop

Setting up Hadoop environment variables

export HADOOP_INSTALL=/usr/local/hadoop

Then set the conf directory

export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop

Answer 2

From /etc/spark/conf/spark-env.sh:

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/etc/hadoop/conf}