All geek questions in one place

Missing application resource while running script in pyspark

I am trying to execute pyspark script.py, but I keep getting this error:

11:55 $ ./bin/spark-submit --jars spark-cassandra-connector-2.0.0-M2-s_2.11.jar --py-files example.py
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)

I can easily execute it by following these steps:

 11:57 $  pyspark --jars spark-cassandra-connector-2.0.0-M2-s_2.11.jar

then paste the block of code block by block into IPython(interactive shell). but I want to put the script in a cronjob so that it can run automatically. I need a command to enter cronjob, but it spark-submitdoesn’t work. Any ideas?

+6
source share
1 answer

you need to put the python file at the end again.

./bin/spark-submit --jars spark-cassandra-connector-2.0.0-M2-s_2.11.jar --py-files example.py example.py
+1
source

Source: https://habr.com/ru/post/1017166/

More articles:


All Articles