HDPCD SPARK - WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources


#1

Hi,

I am working on exercise as part of HDPCS SPARK using python. I am able to start the spark session using pyspark and create RDD collection orders, however when trying to execute the first method of collection it throws below error

WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Below are the list of commands i have executed on hortonworks sandbox running locally.

Please guide me to resolve this issue.

from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster(“yarn-client”).setAppName(“Testing”).set(“spark.ui.port”,“12356”)
sc = SparkContext(conf=conf)
orders = sc.textFile("/user/root/data/retail_db/orders")
orders.first()
18/03/12 22:36:58 INFO FileInputFormat: Total input paths to process : 1
18/03/12 22:36:58 INFO SparkContext: Starting job: runJob at PythonRDD.scala:393
18/03/12 22:36:58 INFO DAGScheduler: Got job 0 (runJob at PythonRDD.scala:393) with 1 output partitions
18/03/12 22:36:58 INFO DAGScheduler: Final stage: ResultStage 0 (runJob at PythonRDD.scala:393)
18/03/12 22:36:58 INFO DAGScheduler: Parents of final stage: List()
18/03/12 22:36:58 INFO DAGScheduler: Missing parents: List()
18/03/12 22:36:58 INFO DAGScheduler: Submitting ResultStage 0 (PythonRDD[2] at RDD at PythonRDD.scala:43), which has no missing parents
18/03/12 22:36:58 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.8 KB, free 252.1 KB)
18/03/12 22:36:58 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.0 KB, free 255.1 KB)
18/03/12 22:36:58 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.2.15:58749 (size: 3.0 KB, free: 511.5 MB)
18/03/12 22:36:58 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
18/03/12 22:36:58 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (PythonRDD[2] at RDD at PythonRDD.scala:43)
18/03/12 22:36:58 INFO YarnScheduler: Adding task set 0.0 with 1 tasks
18/03/12 22:37:13 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources