Clarification regarding spark executor memory


Dear Team,

I was launching spark CLI like below (on

[rrajkumar9999@gw03 ~]$ spark-shell --master yarn --num-executors 25 --conf spark.port.ui=55555

Understanding is that our lab default
Executor memory is 1 GB
Driver memory is 512 MB
Yarn AM memory is 512 MB

But able to see below message during spark shell launch.
18/04/10 03:55:32 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
18/04/10 03:55:35 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
18/04/10 03:55:40 INFO BlockManagerMasterEndpoint: Registering block manager with 511.1 MB RAM, BlockManagerId(driver,, 35266)

My understanding is that each executor will be created with size 1 GB + 384 MB. Why it is being created with size 511.1 MB

Thanks in advance.


The memory that is reported is driver memory and the default memory is only 512MB and overhead memory 384MB. YARN AM memory in our cluster is 1 GB by default and it is not reported here.

Driver memory from /etc/spark/conf/

#SPARK_DRIVER_MEMORY="512M" #Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb)

Overhead memory from /etc/spark/conf/spark-defaults.conf

spark.yarn.driver.memoryOverhead 384

We can also run spark jobs using spark-submit in yarn cluster mode using --deploy-mode, in which case it take 1024 MB + 384 MB for the container as driver program runs as part of application master.

18/04/10 08:18:32 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead

The values are from our state of the art Big Data cluster with Spark. Please visit to sign up to the cluster.