Accessing Spark 2.0 on the cluster

@itversity @venkatreddy-amalla, how to access Spark 2.0 while launching pyspark shell.

Hi All,

I have found out a way to launch Spark 2.0 shell in bigdata-labs.com, you can find the pyspark file at

/usr/hdp/current/spark2-client/bin/pyspark

If you would like to launch spark2.0 files using spark-submit command, the spark-submit command can be accessed using

/usr/hdp/current/spark2-client/bin/spark-submit

and then specify your .py file and properties.

@itversity, SPARK_MAJOR_VERSION property is not set and since multiple versions of Spark are installed on the cluster, Spark 1.6 is being run by default.

Happy coding on bigdata-labs.com

2 Likes

@pramodvspk Nice Answer :slight_smile: . Just wanted to let you people know another easy way to access Spark2 spark-shell (not spark-submit).

Execute the following command before your run spark-shell or pyspark:

export SPARK_MAJOR_VERSION=2

Note: This works with spark-shell/pyspark. If you want to submit jobs on spark2 cluster you need to follow the steps provided by @pramodvspk.

2 Likes

Hi ,

Iam facing below error while creating jar file in STS using with the help of sbt plugins install in window OS .

16/12/01 13:40:05 INFO deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
16/12/01 13:40:06 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.(Shell.java:293)

Any one have any solution for this?.
thanks

I am following the above video and trying to execute spark using Scala IDE and SBT but getting the similar problem(winutils.exe)

Do I need to install hadoop binaries in Windows too (to execute the program)?

17/02/11 17:55:33 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.(Shell.java:293)