Spark verison in lab - 1.6.2

pyspark

#1

Hi,
In tehe Recent spark videos, durga sir explains the spark using the version 2.3.0 where as in the lab it is still 1.6.2. Is there a way to switc the spark version in the lab to 2.3.0?. If yes, please let me know.

Thanks,
Subbu


#2

Try this:

export SPARK_MAJOR_VERSION=2
spark-shell
–master yarn
–deploy-mode client
–conf spark.ui.port=12335
–num-executors 2
–executor-memory 512M


#3

Thanks. It works. Will come back, if needed any further help