Apache Spark 1.6 - Transform, Stage and Store - Setup Spark on Windows

As part of this topic, we will set up Spark in Windows Machine

  • Go to the link Spark Download
  • Choose the relevant version of Spark(this time it is 1.6.3)
  • Choose the Hadoop version(this time it is 2.6)
  • Click on the download link which will take to the downloads page
  • Copy the link and paste it in the Cygwin
  • wget http://www-us.apache.org/dist/spark/spark-1.6.3/spark-1.6.3-bin-hadoop2.6.tgz
  • Copy the Spark folder from Cygwin to Local Directory
  • Download winutils from here , save the link into your local machine by creating a hadoop/bin folder
  • Set the path in the system variables
  • Change the permissions if still the spark-shell is not launched
  • Set the Spark path in system variables

Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster