Exception while running spark-submit

apache-spark

#1

I am running the below spark-submit and it throws exception saying “java.lang.ClassNotFoundException: New_Project”

spark-submit --class New_Project
–master yarn
–deploy-mode client
–conf spark.ui.port=12456
hdfs://nn01.itversity.com:8020/user/mraheemabdul/New_Project.jar

Code Snippet:

import org.apache.spark.sql.SparkSession

object SimpleApp {
def main(args: Array[String]) {
val logFile = “/user/mraheemabdul/README.md”
val spark = SparkSession.builder.appName(“Simple Application”).getOrCreate()
val logData = spark.read.textFile(logFile).cache()
val numAs = logData.filter(line => line.contains(“a”)).count()
val numBs = logData.filter(line => line.contains(“b”)).count()
println(s"Lines with a: $numAs, Lines with b: $numBs")
spark.stop()
}
}

SBT:

name := “New_Project”

version := “1.0”

scalaVersion := “2.11.0”

libraryDependencies += “org.apache.spark” %% “spark-core” % “2.3.0”
libraryDependencies += “org.apache.spark” %% “spark-sql” % “2.3.0”


#2

Your project structure is not correct create a directory with any name first and then go inside the directory and then create build.sbt file.
The program should be created inside the path /directoryname/src/main/scala/program.scala
you can steps mentioned in this topic below


#3

@Sunil_Itversity

I created this application on Intellij and converted into jar file and shipped the jar file onto my home directory and have been running the spark-submit command.

Please correct me if this is wrong or am i missing on anything.


#4

Then check the path once if you are using windows then the path do not start with user


#5

i saved the readme.md in my itversity home directory.
i am not trying to run the program on Intellij, instead i build the jar file shipped onto itversity cluster and also the datafile and then running the spark-submit command


#6

I have created a demo1 project in your account follow that method to develop the application and run it on the cluster use following command to run the jar file deploy mode must be the cluster not the client.

spark-submit \
--class SimpleApp \
--master yarn \
--deploy-mode cluster \
--conf spark.ui.port=12456 \
hdfs://nn01.itversity.com:8020/user/mraheemabdul/demo1_2.11-1.0.jar