How can I upgrade to spark 2.4 from 1.6

Hold on, read this through before raising topic in this category

Are you getting Permission denied, too many logins issue?
Don’t raise new ticket. Click here for the solution. If the issue persists after 30 minutes then raise new ticket

Go through other common issues in this category before raising any issue.

Hi @Anil4,

while launching spark shell use pyspark2 (for spark-with-python) or spark2-shell(for spark-with scala) with your command ex-

pyspark2 --master yarn --conf spark.ui.port=0

spark2-shell --master yarn --conf spark.ui.port=0

OR update your bash profile:

vi ~/.bash_profile

after this add below line


save and exit from here, then re-login to lab.

1 Like