How to run a script in PySpark

Question

I am trying to run a script in a pyspark environment, but so far I have not been able to. How can I run a script like python script.py but in pyspark? Thanks

Answer 1

You can do: ./bin/spark-submit mypythonfile.py

Running python applications through pyspark not supported in Spark 2.0.

Answer 2

pyspark 2.0 and later execute the script file in the environment variable PYTHONSTARTUP , so you can run:

 PYTHONSTARTUP=code.py pyspark 

Compared to spark-submit answer, this is useful for running initialization code before using the pyspark interactive shell.

Answer 3

Just spark-submit mypythonfile.py should be enough.