Unable to save file as snappy after gzip


#1

Dear All,
while solving aruns blog.I faced one problem . I want to save dataframe as gzip and snappy.I have written following code.

  1. sqlContext.setConf(“spark.sql.parquet.compression.codec”,“gzip”)
    result.write.parquet(“path”)

  2. sqlContext.setConf(“spark.sql.parquet.compression.codec”,“uncompressed”)

  3. sqlContext.setConf(“spark.sql.parquet.compression.codec”,“snappy”)
    result.write.parquet(“path”)

after first step result show gz.parquet files as expected but after step 3 also it shows same gz.parquet files ,Even I uncompressed it still it is not showing snappy .Please help


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster