Spark.sql.shuffle.partitions does not work

Spark.sql.shuffle.partitions does not work
4.0 1


Even after setting this as 2 (spark.sql.shuffle.partitions) , it creates 200 output files .

scala> sqlContext.getConf(“spark.sql.shuffle.partitions”)
res34: String = 2


You have to set no of partitions by


I did that and then only i checked the value using getConf.

One more question .
Do we need to import avro packages in cloudera to use it as below .

spark-shell --packages com.databricks:spark-avro_2.10:2.0.1 --master yarn --conf spark.ui.port=5611


Here is my shell output:

scala> sqlContext.getConf(“spark.sql.shuffle.partitions”)
res0: String = 200

scala> sqlContext.setConf(“spark.sql.shuffle.partitions”,“2”)

scala> sqlContext.getConf(“spark.sql.shuffle.partitions”)
res2: String = 2

Yes you have to import by
import com.databricks.spark.avro._


Simple hive subquery is not working . Do you know why .

FAILED: ParseException line 1:24 cannot recognize input near ‘inner’ ‘.’ ‘order_date’ in selection target

select inner.order_date from (select Y.order_date, count(1) as total_orders from orders_sqoop as Y group by Y.order_date order by total_orders desc, Y.order_date desc limit 1) inner


inner is keyword. Don’t use predefined sql keyword as alias. Use another word like innerTable some thing like that. There is mistake in arun’s sir blog.