Issue with lzo codec

I am getting error while trying lzo codec… please advice. Tnx

scala> sqlContext.setConf(“spark.sql.parquet.compression.codec”,“lzo”)

scala> sqlContext.getConf(“spark.sql.parquet.compression.codec”)
res38: String = lzo

scala> ordersByStatus.write.parquet("/user/cloudera/testFolder/orders_parquet_compresslzo")
[Stage 71:===================================================> (11 + 1) / 12]17/06/06 20:14:44 WARN spark.ExecutorAllocationManager: No stages are running, but numRunningTasks != 0
17/06/06 20:14:45 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 75.0 (TID 275, 172.16.206.154): parquet.hadoop.BadConfigurationException: Class com.hadoop.compression.lzo.LzoCodec was not found

To use lzo cluster need to be configured with it. By default lzo need not be available in many clusters.

1 Like