Clarification regarding spark executor memory


#1

Dear Team,

I was launching spark CLI like below (on https://labs.itversity.com).

[rrajkumar9999@gw03 ~]$ spark-shell --master yarn --num-executors 25 --conf spark.port.ui=55555

Understanding is that our lab default
Executor memory is 1 GB
Driver memory is 512 MB
Yarn AM memory is 512 MB

But able to see below message during spark shell launch.
18/04/10 03:55:32 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
18/04/10 03:55:35 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
18/04/10 03:55:40 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.1.113:35266 with 511.1 MB RAM, BlockManagerId(driver, 172.16.1.113, 35266)

My understanding is that each executor will be created with size 1 GB + 384 MB. Why it is being created with size 511.1 MB

Thanks in advance.
Raj.


#2

The memory that is reported is driver memory and the default memory is only 512MB and overhead memory 384MB. YARN AM memory in our cluster is 1 GB by default and it is not reported here.

Driver memory from /etc/spark/conf/spark-env.sh

#SPARK_DRIVER_MEMORY="512M" #Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb)

Overhead memory from /etc/spark/conf/spark-defaults.conf

spark.yarn.driver.memoryOverhead 384

We can also run spark jobs using spark-submit in yarn cluster mode using --deploy-mode, in which case it take 1024 MB + 384 MB for the container as driver program runs as part of application master.

18/04/10 08:18:32 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead


The values are from our state of the art Big Data cluster with Spark. Please visit https://labs.itversity.com to sign up to the cluster.