YarnClientSchedulerBackend inserting messages in pyspark console


#1

Hello,

While working through videos in cca-175-spark-and-hadoop-developer-python course and signed into “https://labs.itversity.com/user/console”, I keep getting random messages from YarnClientSchedulerBackend inserted into the console. The pyspark shell is not idle for even 1 second and when the random message is inserted by YarnClientSchedulerBackend in the console, it totally messes up the script which I am writing which is very annoying. Please see the image attached. This keeps happening and am looking for advice on how to turn this off. Could you please give me some advice ?

Thank you.

Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster


#2

Any response for this question ?


#3

You just need to use sc.setLogLevel(“ERROR”) to suppress these log messages.


#4

Thank you Durga for the quick response.