CCA175:Configuration: increase available memory

The official site of CCA175: https://www.cloudera.com/more/training/certification/cca-spark.html mentions that we have to set the configuration: “Supply command-line options to change your application configuration, such as increasing available memory”

What is the command to do it? Does anyone know it?

Thanks,
Ramya

There are several arguments which can be passed while submitting spark applications or launching spark-shell/pyspark

./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://207.184.161.138:7077 \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000

All these are important parameters

–num-executors
–executor-cores
–executor-memory
–total-executor-cores

Here are few important links

3 Likes

But how do we run these commands? Where on command prompt? Just open a command prompt window and run this above command? But that is not working.

These are the arguments that could be passed while submitting the spark applications using spark-submit as shown in the post, else you can set these arguments while launching the spark-shell like
spark-shell --master yarn-client --executor-memory 2g