After reading textFile How to create RDD to DF with out creating Case class


#1

After reading texFile , how can we convert RDD to DF without creating case class

val textData=sc.textFile(“read orders textFile location”)

textData.map(x=>x.split("|)).map(rec.orders(rec(0).toInt,rec(1).toInt,rec(2).toString,rec(3).toString)).toDF(“orderid”,“order_amt”,“order_status”,“order_date”);

Can any one pls suggest the correct way ,


#2

Hi @ethabhi2795,

Check the file path you used. It’s just text.

Please use the code snippet given below to see how to map fields a Data Frame.

val orders = sc.textFile("/public/retail_db/orders")

orders.map(x => (x.split(",")(0).toInt,x.split(",")(1),x.split(",")(2).toInt,x.split(",")(3))).toDF(“orderId”,“date”,“customerId”,“status”).show()

If you still have queries Let me know.

result will be like the below image:
Screenshot%20from%202018-03-22%2013-01-30

thanks,
Padmavathi


#3

@ethabhi2795 If I understand your question correctly, we can’t create the expected layout of DF without a model (Case Class or Singleton Object or SturctFields) else you might ended up with one column or _1, _2, _3 etc).


#4

@email2dgk to create a DF we can create DF without creating Case class like below . this will save time during exam time.

orders.map(x => (x.split(",")(0).toInt,x.split(",")(1),x.split(",")(2).toInt,x.split(",")(3))).toDF(“orderId”,“date”,“customerId”,“status”).show()
[/quote]


#5

@ethabhi2795 Agreed if its for exam but

orders.map(x => {val z = x.split(","); (z(0).toInt, z(1), z(2).toInt}).toDF(“orderId”,“date”,“customerId”,“status”).show()