Launching spark-Shell not working


#1

Launch Command
spark-shell --master yarn
–conf spark.ui.port=12754
–num-executors 1
–executor-memory 512

Exception while loading Spark-Shell

Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_e25_1520592249193_20001_01_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:933)
at org.apache.hadoop.util.Shell.run(Shell.java:844)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1123)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:225)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

18/03/20 10:44:07 INFO BlockManagerMaster: Removal of executor 1 requested
18/03/20 10:44:07 INFO YarnClientSchedulerBackend: Asked to remove non-existent executor 1
18/03/20 10:44:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
18/03/20 10:44:17 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_e25_1520592249193_20001_01_000003 on host: wn04.itversity.com. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_e25_1520592249193_20001_01_000003
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:933)
at org.apache.hadoop.util.Shell.run(Shell.java:844)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1123)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:225)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

18/03/20 10:44:17 INFO BlockManagerMaster: Removal of executor 2 requested
18/03/20 10:44:17 INFO BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster.
18/03/20 10:44:17 INFO YarnClientSchedulerBackend: Asked to remove non-existent executor 2
18/03/20 10:44:20 INFO DiskBlockManager: Shutdown hook called
18/03/20 10:44:20 INFO ShutdownHookManager: Shutdown hook called


#2

@hi2rajnish Use this command
spark-shell --master yarn --conf spark.ui.port=12754 --num-executors 1 --executor-memory 512M


#3

same command when I ran after some time it started, but on that day it was happening frequently


#4

@hi2rajnish You missed M in your command at executor memory. it caused the error.


#5

many thanks that was silly mistake


#6