Connect to the spark screen through the SOCKS proxy server

Question

TL DR . How to connect a local driver to a spark switch through a SOCKS proxy.

We have a spark cluster in place that sits behind a firewall that blocks most ports. We have ssh access, so I can create a SOCKS proxy with ssh -D 7777 ...

It works great for viewing the web interface when my browser uses a proxy server, but I do not know how to use its local driver.

So far I have had this, which obviously does not configure the proxy:

 val sconf = new SparkConf() .setMaster("spark://masterserver:7077") .setAppName("MySpark") new SparkContext(sconf) 

What information for these messages is sent 16 times before throwing an exception.

 15/01/20 14:43:34 INFO Remoting: Starting remoting 15/01/20 14:43:34 ERROR NettyTransport: failed to bind to server-name/ip.ip.ip.ip:0, shutting down Netty transport 15/01/20 14:43:34 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 15/01/20 14:43:34 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 15/01/20 14:43:34 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 15/01/20 14:43:34 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down. 

Answer 1

Your best setMaster("spark://localhost:nnnn") to forward the local port to the remote 7077 and then setMaster("spark://localhost:nnnn") , where nnnn is the local port that you forwarded.

To do this, use ssh -L (instead of -D ). I cannot guarantee that this will work or if it works, that it will continue to work, but at least it will save you from using the actual proxy for this single port. Things that can break it are mostly secondary compounds that can initiate an initial connection. I have not tested this yet, but if there are no secondary connections, in principle this should work.

Also, this does not answer the TL-DR version of your question, but since you have SSH access, it is more likely to work.