How to use sortBy key correctly

Big Data Apache Spark
#1

Hi All,

While implementing sort By Key operation i am getting some error.
It is my sample data.
scala> productsMap.take(4).foreach(println)
(2,(Q64,59.98))
(2,(Armour,129.99))
(2,(Armour,89.99))
(2,(Armour,89.99))
(Product_department_id , (products_name , price ) )

val productsMapGroup = productsMap.groupByKey().map(x => ( x._2.toList.sortBy( k => k.split(",")(1)) ) )
:31: error: value split is not a member of (String, Float)
** val productsMapGroup = productsMap.groupByKey().map(x => ( x._2.toList.sortBy( k => k.split(",")(1)) ) )**

Up to my understanding , (x => ( x._2.toList.sortBy( k => k.split(",")(1)) ) ) the variable x is ((products_name , price ) ) since we used x._2 tuple notation but we have turned the tuple into list by type casting it .
The error show it is still a list so i tried this by still considering it a list and used below command .

val productsMapGroup = productsMap.groupByKey().map(x => ( x._1 , x._2.toList.sortBy( k => k._2) ) )
productsMapGroup: org.apache.spark.rdd.RDD[(String, List[(String, Float)])] = MapPartitionsRDD[10] at map at :31

The Command ran successfully and the result is correct .
Kindly let me know the problem in it.

#2

Hi Pradeep,

Since your value part in productsMap is in itself a key-value pair, split is not required and would not work. Please try the below:

val productsGroupBy = productsMap.groupByKey().flatMap(x => x._2.toList.sortBy(k => k._2.toFloat)).take(100).foreach(println)

Not sure why the underscore symbol is not displayed (x => x.underscore2 and k => k.underscore2) in the command above
Thanks.

#3

Hi santy
Thank you for quick response .
I have tried this earlier but i got the error as you can see that i have used .toList method below.
val productsMapGroup = productsMap.groupByKey().map(x => ( x._2.toList.sortBy( k => k.split(",")(1)) ) )
:31: error: value split is not a member of (String, Float)
The error is still showing this as an tuple and when i have tried this as considering tuple, it worked.
val productsMapGroup = productsMap.groupByKey().map(x => ( x.1 , x.2.toList.sortBy( k => k._2) ) )
Here my concerns is am i making mistake in using toList ?
In my map function .map(x => ( x._2.toList.sortBy( k => k.split(",")(1)) ) ) i have tried to convert it into List but it is still considering it to int .
I have already converted price to float in earlier map operations.