I have file with header and I want to remove that. So I’m done code like below.
val file = sc.readTest("s3://…);
val header = rrd.first();
val data = file.filter(_ 1= header);
and above code working good.
But doubt is spark always read data in order which file order. if yes my solution is correct, if not what is the solution and I dont want solution like val data = file.filter(“col1,col2…”= header);