How to run a script in Pyspark and then put it in an IPython shell?

Question

I want to run a spark script and go to the IPython shell to interactively explore the data.

Running both:

$ IPYTHON=1 pyspark --master local[2] myscript.py 

and

 $ IPYTHON=1 spark-submit --master local[2] myscript.py 

both exiting IPython after completion.

It seems very simple, but cannot find how to do it anywhere.

Answer 1

If you run the iPython shell using

 $ IPYTHON=1 pyspark --master local[2] 

You can do:

  >>> %run myscript.py 

and all variables will remain in the workspace. You can also debug step by step:

 >>> %run -d myscript.py 

Answer 2

Launch the IPython shell using IPYTHON=1 pyspark , then run execfile('/path/to/myscript.py') , which should run your script inside the shell and return to it.