All geek questions in one place

Setting environment variables from python code for spark

I set environment variables on Mac os to run pyspark

 export SPARK_HOME=/Users/devesh/Downloads/spark-1.5.1-bin-hadoop2.6 export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH

The above lines work for me.

I am trying to replicate the above commands in python using the following lines of python code

 os.environ['SPARK_HOME']="/Users/devesh/Downloads/spark-1.5.1-bin-hadoop2.6" spark_home=os.environ.get('SPARK_HOME',None) sys.path.append("/Users/devesh/Downloads/spark-1.5.1-bin-hadoop2.6/python/") sys.path.insert(0,os.path.join(spark_home,'py4j-0.8.2.1-src.zip'))

but it does not work. Please tell me what am I doing wrong?

+1
source share
1 answer

Last line of python code:

 sys.path.insert(0,os.path.join(spark_home,'py4j-0.8.2.1-src.zip'))

does not match your shell code. You can change it to:

 sys.path.insert(0,os.path.join(spark_home,'python','lib','py4j-0.8.2.1-src.zip'))
+1
source

Source: https://habr.com/ru/post/1012061/

More articles:


All Articles