Spark latest versions

Spark latest versions
5.0 1

#1

Hello, Can we use versions other than 1.6.2 (2.0+) here? If yes, how?


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster


#2

@Vihit_Shah

The default version of spark in labs is 1.6.2 if you want to use spark2 then you need to use the below command before launching the spark-shell

export SPARK_MAJOR_VERSION=2
spark-shell \
--master yarn \
--deploy-mode client \
--conf spark.ui.port=12335 \
--num-executors 2 \
--executor-memory 512M