Im facing some issues with pyspark after running a scripts

pyspark

#1

18/10/09 23:11:14 INFO YarnClientSchedulerBackend: Requesting to kill executor(s) 2
18/10/09 23:11:14 INFO ExecutorAllocationManager: Removing executor 2 because it has been idle for 10 seconds (new desired total will be 1)


#2

Hi @rayudu_darapaneni Can you paste the command that you are using.


#3

In general, whatever the command i run its getting into some issues always. For example,
products=sc.textFile(’/public/retail_db/products’)
for i in products.take(10):print(i)
productsMap=products.
filter(lambda x: x.split(’,’)[4] != ‘’).
map(lambda x: (float(x.split(’,’)[4]),x))
productsMapSort=productsMap.sortByKey()
for i in productsMapSort.take(10):print(i)


#4

The command is correct and it is working fine.


#5

yes, that’s true. But lot of unnecessary things are happening after i run any command. Like killing jobs and so on…


#6

Use sc.setLogLevel(“ERROR”) to turn off log infos after launching pyspark