I am working on spark and using notebook on Itversity labs for practice. I am unable to find IDE link, kindly confirm how can I access IDE. In case if IDE is not available, How can I access Spark UI from notebook.
The way we can see it is:
If you are running a yarn job we can get these details from Resource Manager.
Or else you can get details from the respective version of Spark History UI.
Spark2History - http://m01.itversity.com:18081/
Spark3History - http://m02.itversity.com:18080/
As it is spark 2.4 click on Spark2History - http://m01.itversity.com:18081/
If that is completed job using spark user identify your application id. Click on it.
If the job is still running scroll down the page click on show incomplete applications.
Check for user name & click on the spark job application ID you want to see.
I hope it helps.