Itversity labs have two versions of Spark. The default version of spark is 1.6.2 and if you want use spark2 you need to use below command before launching spark-shell
export SPARK_MAJOR_VERSION=2 spark-shell \ --master yarn \ --deploy-mode client \ --conf spark.ui.port=12335 \ --num-executors 2 \ --executor-memory 512M
Common issues we encounter on our state of the art Big Data Cluster with Hadoop, Spark and many others - https://labs.itversity.com
This is to simplify our support process so that we can answer technical issues as well.