Does Mode (Yarn / local) matters while submitting Spark Job

apache-spark
pyspark
#1

Does the spark submit mode depends on where the result is getting stored?
i.e., Running in Yarn mode for HDFS destination and Local to save in local FS

I tried to submit the python script which saves result in local FS. spark-submit works well in local mode. But when tried the same in yarn mode, it throws IOException saying Mkdirs failed to create some temp files in output directory

To save the results in local file system is it mandatory to submit spark job in local.

0 Likes

#2

Yes, you will not be able to save to local file system using YARN mode.

0 Likes

#3

Thanks Durga.:slight_smile:

0 Likes