HDPCD-Spark Run a jar using spark-submit in yarn mode


#1

Hi Team,
This in regards to a spark certification(PYTHON) question. To run a JAR file using Spark-Submit in YARN mode.

  1. Will we be given the JAR file name & location,
  2. Will they also provide the value for --class argument
  3. please pass a working example or a .JAR file to execute, i was not successful running

./bin/spark-submit --class org.apache.spark.examples.SparkPi
–master yarn
–deploy-mode cluster
–driver-memory 2g
–executor-memory 2G
–executor-cores 4
–queue default
/usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.5.0-292-hadoop2.7.3.2.6.5.0-292.jar
10


Prepare for certifications on our state of the art labs which have Hadoop, Spark, Kafka, Hive and other Big Data technologies

  • Click here for signing up for our state of the art 13 node Hadoop and Spark Cluster


#2

@Prem1,

  1. They will give all the details.
    To run, SparkPi, Use the below command

cd/bin
Then,

spark-submit --master yarn --deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--driver-memory 2G \
--executor-memory 2G \
--executor-cores 4 \
/usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.5.0-292-hadoop2.7.3.2.6.5.0-292.jar