Intermittent messages on the iteversity cluster

Big Data Apache Spark
#1

I see messages like below from the cluster from time to time.

What are these? Are these related to my session? I dont think I have declared any accumulators or broadcast variables. Could somebody please explain why I see these.

Thanks.

>>> 16/12/26 16:28:39 INFO BlockManagerInfo: Removed broadcast_8_piece0 on localhost:44285 in memory (size: 3.5 KB, free: 511.0 MB)
16/12/26 16:28:39 INFO ContextCleaner: Cleaned accumulator 5
16/12/26 16:28:39 INFO BlockManagerInfo: Removed broadcast_6_piece0 on localhost:44285 in memory (size: 3.4 KB, free: 511.0 MB)
16/12/26 16:28:39 INFO ContextCleaner: Cleaned accumulator 4
16/12/26 16:28:39 INFO BlockManagerInfo: Removed broadcast_4_piece0 on localhost:44285 in memory (size: 3.4 KB, free: 511.0 MB)
16/12/26 16:28:39 INFO ContextCleaner: Cleaned accumulator 3
16/12/26 16:28:39 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:44285 in memory (size: 28.4 KB, free: 511.0 MB)
16/12/26 16:28:39 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:44285 in memory (size: 3.4 KB, free: 511.0 MB)
16/12/26 16:28:39 INFO ContextCleaner: Cleaned accumulator 2
#2

These are called verbose messages, these are used to understand what’s happening in the background after we run any transformations and actions in spark-shell. If these are distracting you, please refer below link, Are Log Messages too Verbose in spark shell, then fix it.

#3

Thank you Ravi… But why would I get these messages when I am not running anything. I get these even when I leave my session open for a while. Any thoughts?

#4

Spark-shell is an interactive shell, it is build with scala language. Scala language has shared variables are of 2 types, Broadcasters and accumulators.As far as my knowledge, the interactive session with these shared variables keep running in background. When you have invoked a spark-shell then observe in Hue Job browser, then you can see that spark-shell job is running. You can see the properties and resources of that job to find clear stats about it.

If i got any more depth internals about spark-shell will share in this window.

2 Likes
#5

That gave me a lot of insight… Thank you…