Cannot read parquet file in dataframe

dataframes

#1

i have a compressed parquet file in hdfs.
I need to read the file for aggregation, but it is throwing error.

org.apache.spark.sql.AnalysisException: Parquet type not yet supported:
INT64 (TIMESTAMP_MILLIS);

code used:
val n = sqlContext.read.parquet(“administration_set.parquet.gzip”)

I found that spark supports the INT64 (TIMESTAMP_MILLIS) type. However,it’s after the 2.3 version,

Do we have any work around for spark 1.6

please suggest


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster