Dataframe creation

I am using Spark version 1.6.3. I am trying to create a dataframe with the below command. But it is erroring out . Can you please let me know if i am missing any imports.

Command used

val test= spark.read.option(“inferSchema”,“true”).option(“header”,“true”).csv("/user/x1871688/test34/
user.csv")

:33: error: not found: value spark
val test=spark.read.option(“inferSchema”,“true”).option(“header”,“true”).csv("/user/x1871688/test34/user.csv")


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster

hi you need to specify csv inside format and use load to load file as below:

val test= spark.read.format(“csv”).option(“inferSchema”,“true”).option(“header”,“true”).load("/user/sri_karthik/kv2.csv")