Do i need to use /bin/pyspark mentioned in youtube videos

Do i need to use /bin/pyspark to initiate pyspark shell when i use bigdata labs as mentioned in youtube video for Cloudera VM

You just need to type pyspark

so no need to make spark 1.2.1 ?

@Tarun_Das that depends on which version of Cloudera Quickstart VM you are using.

actually what confused me that video says now we can ignore set up of spark 1.2.1 as data frames no longer needed in certification

Yes, if you are preparing for CCA175 certification then you need to practice and provide exam on Spark 1.2.1. In the videos Durga sir has demonstrated videos in Cloudera Quickstart VM 5.4 in which it has been built with Spark 1.3.0. The technical features differences between these 2 versions are DataFrames added in Spark 1.3.0. So if you don’t want to configure for 1.2.1 in cloudera, then you can use Spark 1.3.0 as long as you are not using DataFrames.

P.S: DataFrames is out of scope for certification.

1 Like

Thank you for the information