Error while running wordcount program in yarn mode

Error while running wordcount program in yarn mode
5.0 1


I am trying to run the word count program in yarn mode. I have tried two sets of commands as of now. The commands are listed below:

  1. spark-submit --master yarn --conf spark.ui.port=43679 --class wordcount.WordCount sparkdemo_2.10-1.0.jar prod /public/randomtextwriter /Users/ashwinvprabhu/wordcount

  2. spark-submit --class wordcount.WordCount --master yarn --conf spark.ui.port=43679 sparkdemo_2.10-1.0.jar prod /public/randomtextwriter /Users/ashwinvprabhu/wordcount

Both of these commands give me the below output:

Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
Warning: Local jar /home/ashwinvprabhu/sparkdemo_2.10-1.0.jar does not exist, skipping.
java.lang.ClassNotFoundException: wordcount.WordCount
at java.lang.ClassLoader.loadClass(
at java.lang.ClassLoader.loadClass(
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(
at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

From this error, it is clear that the spark-submit looks for the jar file at “/home/ashwinvprabhu/sparkdemo_2.10-1.0.jar”? How do I fix this error?


I solved the issue. Uploaded the files to Ambari and then copied it to local file system using hadoop copyToLocal command. The program is now running but exits when it tries to create the wordcount output folder (Permission denied for write).


@ashwinvprabhu Give the output location like /user/ashwinvprabhu/output_file_name and check. Paste the command and error if you still facing the problem. Mark as solved if you got output.