Error on submitting spark job

apache-spark
scala

#1

Hi,
I’ve developed app jar at Intellij idea on windows platform and shipped to hadoop cluster thru winscp. The jar file location is under: /user/saikiranchepuri35/FirstSparkApp.jar

I’ve used following syntax:
spark-submit --master yarn --class FirstSparkApp --deploy-mode cluster
–executor-memory 1g hdfs://nn01.itversity.com:8020/user/saikiranchepuri35/FirstSparkApp.jar
(I’ve not created any package for main class, so Ive given directly)

When I try to run, it was failing:
18/10/26 07:16:12 ERROR ApplicationMaster: Uncaught exception:
java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:284)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:238)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
at java.util.jar.JarVerifier.update(JarVerifier.java:228)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:383)
at java.util.jar.JarFile.getInputStream(JarFile.java:450)
at sun.misc.JarIndex.getJarIndex(JarIndex.java:137)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:839)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:831)
18/10/26 07:16:12 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes)
18/10/26 07:16:12 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes)
18/10/26 07:16:12 INFO ApplicationMaster: Deleting staging directory .sparkStaging/application_1540458187951_0496
18/10/26 07:16:12 INFO ShutdownHookManager: Shutdown hook called

Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1540552566047
final status: FAILED
tracking URL: http://rm01.itversity.com:19288/cluster/app/application_1540458187951_0496
user: saikiranchepuri35
Exception in thread “main” org.apache.spark.SparkException: Application application_1540458187951_0496 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1143)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1194)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.

Can someone plz explain on this, help me to resolve out of it.:slight_smile


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster