What is being loaded here?


#1

In the practice, I am running this snippet:

val orders = sc.textFile("/user/paslechoix/retail_db/order_items")
orders.count

I got:
res0: Long = 516594

In the folder /user/paslechoix/retail_db/order_items, I have three different formats and 12 files in total:

[paslechoix@gw01 conf]$ hdfs dfs -ls /user/paslechoix/retail_db/order_items
Found 13 items
-rw-r–r-- 3 paslechoix hdfs 0 2018-01-15 16:39 /user/paslechoix/retail_db/order_items/_SUCCESS
-rw-r–r-- 3 paslechoix hdfs 456557 2018-01-15 16:39 /user/paslechoix/retail_db/order_items/part-m-00000.snappy
-rw-r–r-- 3 paslechoix hdfs 459317 2018-01-15 16:39 /user/paslechoix/retail_db/order_items/part-m-00001.snappy
-rw-r–r-- 3 paslechoix hdfs 458768 2018-01-15 16:39 /user/paslechoix/retail_db/order_items/part-m-00002.snappy
-rw-r–r-- 3 paslechoix hdfs 450824 2018-01-15 16:39 /user/paslechoix/retail_db/order_items/part-m-00003.snappy
-rw-r–r-- 3 paslechoix hdfs 257743 2018-01-15 16:55 /user/paslechoix/retail_db/order_items/part-m-00004.gz
-rw-r–r-- 3 paslechoix hdfs 258577 2018-01-15 16:55 /user/paslechoix/retail_db/order_items/part-m-00005.gz
-rw-r–r-- 3 paslechoix hdfs 259786 2018-01-15 16:55 /user/paslechoix/retail_db/order_items/part-m-00006.gz
-rw-r–r-- 3 paslechoix hdfs 254614 2018-01-15 16:55 /user/paslechoix/retail_db/order_items/part-m-00007.gz
-rw-r–r-- 3 paslechoix hdfs 257731 2018-01-15 16:58 /user/paslechoix/retail_db/order_items/part-m-00008.deflate
-rw-r–r-- 3 paslechoix hdfs 258565 2018-01-15 16:58 /user/paslechoix/retail_db/order_items/part-m-00009.deflate
-rw-r–r-- 3 paslechoix hdfs 259774 2018-01-15 16:58 /user/paslechoix/retail_db/order_items/part-m-00010.deflate
-rw-r–r-- 3 paslechoix hdfs 254602 2018-01-15 16:58 /user/paslechoix/retail_db/order_items/part-m-00011.deflate

Which files are loaded into the RDD?
What is the way to verify how many lines in the files?

Thank you very much.


#2

All the files under the path will be loaded into RDD and count will give the number of lines in all the files.