Spark-shell - Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded

spark-shell

#1

|Application Attempt State:|SCHEDULED|
|Started:|Thu Mar 01 10:57:22 -0500 2018|
|Elapsed:|5mins, 9sec|
|AM Container:|N/A|
|Node:|N/A|
|Tracking URL:|ApplicationMaster|
|Diagnostics Info:|[Thu Mar 01 11:00:45 -0500 2018] Application is added to the scheduler and is not yet activated. Queue’s AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:1024, vCores:1>; Queue Resource Limit for AM = <memory:34816, vCores:1>; User AM Resource Limit of the queue = <memory:34816, vCores:1>; Queue AM Resource Usage = <memory:34816, vCores:28>;|

I am unable to start the spark-shell…console shows the following message
18/03/01 11:03:41 INFO Client: Application report for application_1517228278761_24897 (state: ACCEPTED)
18/03/01 11:03:42 INFO Client: Application report for application_1517228278761_24897 (state: ACCEPTED)…

Have tried starting the spark-shell using both of the following commands
spark-shell --master yarn --conf spark.ui.port=60000 --num-executors 1 --executor-memory 512M
spark-shell --master yarn --conf spark.ui.port=0 --num-executors 1 --executor-memory 512M

please advise.


#2

It is working fine now!!!


#3

Same issue today, please see below

18/03/07 16:07:52 INFO Client:
client token: N/A
diagnostics: [Wed Mar 07 16:07:51 -0500 2018] Application is Activated, waiting for resources to be assigned for AM. Details : AM Partition = <DEFAULT_PARTITION> ; Partition Resource = <memory:172032, vCores:42> ; Queue’s Absolute capacity = 100.0 % ; Queue’s Absolute used capacity = 100.0 % ; Queue’s Absolute max capacity = 100.0 % ;
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1520456871732
final status: UNDEFINED
tracking URL: http://rm01.itversity.com:8088/proxy/application_1520435667063_0180/
user: ganesh1146
18/03/07 16:07:53 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:07:54 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:07:55 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:07:56 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:07:57 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:07:58 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:07:59 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:08:00 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:08:01 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)
18/03/07 16:08:02 INFO Client: Application report for application_1520435667063_0180 (state: ACCEPTED)


#4

Seems to be intermittent… resolved itself a few minutes later

Same issue this morning, please see below

18/03/12 14:04:18 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:18 INFO Client:
client token: N/A
diagnostics: [Mon Mar 12 14:04:17 -0400 2018] Application is Activated, waiting for resources to be assigned for AM. Details : AM Partition = <DEFAULT_PARTITION> ; Partition Resource = <memory:172032, vCores:42> ; Queue’s Absolute capacity = 100.0 % ; Queue’s Absolute used capacity = 99.40476 % ; Queue’s Absolute max capacity = 100.0 % ;
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1520877857273
final status: UNDEFINED
tracking URL: http://rm01.itversity.com:8088/proxy/application_1520592249193_5269/
user: ganesh1146
18/03/12 14:04:19 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:20 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:21 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:22 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:23 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:24 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:25 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:26 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:27 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:28 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:29 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:30 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:31 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)
18/03/12 14:04:32 INFO Client: Application report for application_1520592249193_5269 (state: ACCEPTED)