Getting error while Spark-submit

apache-spark

#1

am trying to submit a spark job but getting below error,

spark-submit --class org.girinath.WordCount \
  --master local \
  --deploy-mode client \
  --executor-memory 1g \
  --conf "spark.app.id=wordcount" \
  hdfs://nn01.itversity.com:8020/user/girinathg/SparkJabWc.jar

Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
Warning: Skip remote jar hdfs://nn01.itversity.com:8020/user/girinathg/SparkJabWc.jar.
java.lang.ClassNotFoundException: org.girinath.WordCount
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

please help me to correct this


#2

please let me know if any update


#3

There are several issues in your spark submit command, master have to be set to yarn, deploy-mode have to be set to cluster and also you have passed incorrect jar file name.

Here is the working command:

spark-submit --class org.girinath.WordCount \
  --master yarn \
  --deploy-mode cluster \
  --executor-memory 1g \
  --conf "spark.app.id=wordcount" \
  hdfs://nn01.itversity.com:8020/user/girinathg/SparkJobWc.jar

User was facing the issue while running on the lab which facilitated us to come up with tested solution. In case you want to accelerate your learning process, please sign up to our state of the art Big Data cluster.



#4

thanks alot … you are great :slight_smile:


#5