Spark submit (2.3) on python cluster cluster

Question

So, now that k8s integrates directly with the spark in 2.3, my light feed from the console is done correctly on the main goblet without any spark master pods, the spark processes all the details of k8s:

spark-submit \
  --deploy-mode cluster \
  --class com.app.myApp \
  --master k8s://https://myCluster.com \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.app.name=myApp \
  --conf spark.executor.instances=10 \
  --conf spark.kubernetes.container.image=myImage \
  local:///myJar.jar

What I'm trying to do is fix-submit via AWS lambda to my k8s cluster. Previously, I used the command directly using the REST API of the spark master (without kubernetes):

request = requests.Request(
    'POST',
    "http://<master-ip>:6066/v1/submissions/create",
    data=json.dumps(parameters))
prepared = request.prepare()
session = requests.Session()
response = session.send(prepared)

And it worked. Now I want to integrate Kubernetes and do it the same way when I send an API request to my kubernetes cluster from python and sparkly process all the details of k8s, ideally something like:

request = requests.Request(
    'POST',
    "k8s://https://myK8scluster.com:443",
    data=json.dumps(parameters))

Is this possible in the Spark 2.3 / Kubernete integral?

Answer 1

, Spark 2.3, Kubernetes.

, :

• Spark Spark, Kubernetes.
• , Kubernetes , .
• , - , - "" API Kubernetes , .

, , , , Spark (). .

- AWS Lambda , , - (, , , Bash AWS Lambda), - , AWS Lambda REST spark-submit. .