Facing problem while saving file without compression (spark-scala)


#2

Dear All I got the answer…:smile:


#1

Dear All,
I have saved dataframe into textfile with GzipCodec compression while solving first problem
code is given below
data.map(x=>(x(0)+"\t"+x(1)+"\t"+x(2))).saveAsTextFile(“path”,classOf[org.apache.hadoop.io.compress.GzipCodec])
above code works fine .When I tried to solve next problem in which I need to save data using hiveQL

val result=hql.sql(“select * from product_replica where product_price>100”)
result.write.parquet(“path”)
then my file shows with gz.parquet extension .later I used sqlContext.setConf(“spark.sql.parquet.compression.codec”,“uncompressed”) again I saved the file but still spark force fully adding gz compression…Please help…How to remove effect of compression after using in saveASTextFile() and saveAsSequenceFile()


#3

@jyoti

That’s good can you share how you resolved your issue too? It would be helpful for others who may come across the same issue.

Regards,
Sunil Abhishek


#4

@jyoti,

Awaiting for your answer :slight_smile:


#5

hql.setConf(“spark.sql.parquet.compression.codec”,“uncompressed”)

I have set the configuration over hiveQL reference