Pyspark not launching using this command pyspark --master yarn -conf spark.ui.port=12888

pyspark not launching using this command pyspark --master yarn -conf spark.ui.port=12888.

Please find attachment

issue resolved with adding --conf instead -conf.
But now, after executing this command, I am not finding the tracking url for spark jobs. could you help?

It is because log level is disabled.

You can got to job ui by using http://HOST:PORT

In your case

  • HOST -> gw03.itversity.com
  • PORT -> Port specified as part of spark.ui.port which is 12888

If you go to browser and type http://gw03.itversity.com:12888, you will go to job ui.

Alternatively you can also go to Resource Manager UI and look out for your jobs.

Resource Manager UI: http://rm01.itversity.com:19088/cluster