I am unable to see web UI for Pyspark.
I tried below command,
pyspark --master yarn --conf spark.ui.port=12888
I didn’t see any message saying service ‘SparkUI’ started at port 12888.
Seems like loglevel INFO is not set, that’s why it is not displaying any info.
At the same time, I have one more query how to set loglevel to INFO before logging to pyspark.
I know how to do it when we log into pyspark.
Kindly assist on this.
Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs
- Click here for access to state of the art 13 node Hadoop and Spark Cluster