Lab support request


#1

kindly assist with command needed to trash the unnecessary print out on my lab console please

s = orders.first()
18/07/30 10:45:33 INFO SparkContext: Starting job: runJob at PythonRDD.scala:393
18/07/30 10:45:33 INFO DAGScheduler: Got job 1 (runJob at PythonRDD.scala:393) with 1 output partitions
18/07/30 10:45:33 INFO DAGScheduler: Final stage: ResultStage 1 (runJob at PythonRDD.scala:393)
18/07/30 10:45:33 INFO DAGScheduler: Parents of final stage: List()
18/07/30 10:45:33 INFO DAGScheduler: Missing parents: List()
18/07/30 10:45:33 INFO DAGScheduler: Submitting ResultStage 1 (PythonRDD[3] at RDD at PythonRDD.scala:43), which has no missing parents
18/07/30 10:45:33 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.9 KB, free 377.8 KB)
18/07/30 10:45:33 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.0 KB, free 380.8 KB)
18/07/30 10:45:33 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 172.16.1.113:44613 (size: 3.0 KB, free: 511.1 MB)
18/07/30 10:45:33 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1008
18/07/30 10:45:33 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (PythonRDD[3] at RDD at PythonRDD.scala:43)
18/07/30 10:45:33 INFO YarnScheduler: Adding task set 1.0 with 1 tasks
18/07/30 10:45:34 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 1)
18/07/30 10:45:36 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (wn03.itversity.com:48981) with ID 2
18/07/30 10:45:36 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, wn03.itversity.com, partition 0,NODE_LOCAL, 2167 bytes)
18/07/30 10:45:36 INFO ExecutorAllocationManager: New executor 2 has registered (new total is 1)
18/07/30 10:45:36 INFO BlockManagerMasterEndpoint: Registering block manager wn03.itversity.com:41857 with 511.1 MB RAM, BlockManagerId(2, wn03.itversity.com, 41857)
18/07/30 10:45:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on wn03.itversity.com:41857 (size: 3.0 KB, free: 511.1 MB)


#2

sc.setLogLevel(“ERROR”)