How can I upgrade to spark 2.4 from 1.6

Hold on, read this through before raising topic in this category

Are you getting Permission denied, too many logins issue?
Don’t raise new ticket. Click here for the solution. If the issue persists after 30 minutes then raise new ticket

Go through other common issues in this category before raising any issue.

Hi @Anil4,

while launching spark shell use pyspark2 (for spark-with-python) or spark2-shell(for spark-with scala) with your command ex-

pyspark2 --master yarn --conf spark.ui.port=0

spark2-shell --master yarn --conf spark.ui.port=0

OR update your bash profile:

vi ~/.bash_profile

after this add below line

export SPARK_MAJOR_VERSION=2

save and exit from here, then re-login to lab.

1 Like