Apache Spark 2.x – Data processing - Getting Started - Review of Setup Steps for Spark Environment

Setup Spark Environment

Let us review the details to setup of Spark Environment. We have already covered how to setup environment earlier.

  • Pre-requisites
    • 64 bit Computer
    • At least 4 GB RAM and enough storage
    • 64-bit Operating System -Windows 10,Linux,Mac etc
    • We would recommend Ubuntu on top of Windows 10.You can either setup using Windows Subsystem for Linux or have virtual machine
  • Setup Process
    • Go to http://spark.apache.org/
    • Download the tarball of your choice
    • Uncompress and untar in your favorite location
    • Make sure to setup environment variable so that you can use commands such as spark-shell,pyspark,spark-submit from anywhere
  • Understand the spark layout

Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster