Date data type using data frame and case class in Spark 1.6


#1

Hi All,

Could you please share some details on how to load date data type using case class and data frame in Spark1.6 (using Scala)

I have input file (text file with tab delimiter) with one column having Mm/dd/yy date data and few other columns present in HDFS

I am trying to load it into a data frame and then to temp table to execute some query

Step 1 : load text file as RDD and do map and split to get columns data
Step 2 : define case class
Step 3: create data frame using above rdd and case class and register as temp table

Could you please share some code of loading above date column to data frame and using above steps using Spark 1.6 (in Scala)

I tried defining the date column as string and then cast using to_date while doing sql on temp table, however even tough there is no error I don’t observe data being sorted on date

Thanks,
Anandh