I was launching spark CLI like below (on https://labs.itversity.com).
[rrajkumar9999@gw03 ~]$ spark-shell --master yarn --num-executors 25 --conf spark.port.ui=55555
Understanding is that our lab default
Executor memory is 1 GB
Driver memory is 512 MB
Yarn AM memory is 512 MB
But able to see below message during spark shell launch.
18/04/10 03:55:32 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
18/04/10 03:55:35 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
18/04/10 03:55:40 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.1.113:35266 with 511.1 MB RAM, BlockManagerId(driver, 172.16.1.113, 35266)
My understanding is that each executor will be created with size 1 GB + 384 MB. Why it is being created with size 511.1 MB
Thanks in advance.