Spark submit issue for python files

#1

Hi ,

I am trying following spark submit command in labs. getting below listed error. Can someone please help with spark submit command for python files.

spark-submit --master yarn
–deploy-mode cluster
–num-executors 2
–executor-cores 1 --executor-memory 2g --driver-memory 2g
–conf “spark.yarn.maxAppAttempts=1”
–files hdfs:///user/sunilkalra/code/product_daily_sale.py

[sunilkalra@gw02 usr]$ spark-submit --master yarn --deploy-mode cluster --num-executors 2 --executor-cores 1 --executor-memory 2g --driver-memory 2g --conf “spark.yarn.maxAppAttempts=1” --files hdfs:///user/sunilkalra/code/product_daily_sale.py
SPARK_MAJOR_VERSION is set to 2, using Spark2
Exception in thread “main” java.lang.IllegalArgumentException: Missing application resource.
at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:253)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:161)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:275)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:152)
at org.apache.spark.launcher.Main.main(Main.java:87)

0 Likes

#2

Can someone please help with it.

0 Likes

#3

I was able to run spark submit.

i have to be in /usr/bin directory to do spark submit.

0 Likes