How to launch spark2 in big data labs?

spark-shell
apache-spark

#1

Itversity labs have two versions of Spark. The default version of spark is 1.6.2 and if you want use spark2 you need to use below command before launching spark-shell

export SPARK_MAJOR_VERSION=2
spark-shell \
--master yarn \
--deploy-mode client \
--conf spark.ui.port=12335 \
--num-executors 2 \
--executor-memory 512M

Common issues we encounter on our state of the art Big Data Cluster with Hadoop, Spark and many others - https://labs.itversity.com

This is to simplify our support process so that we can answer technical issues as well.



How to start spark2 shell
Spark2-shell not working
Spark 2 access issue in labs
#2

Working with Spark 2.0 or higher in the labs