Exercise 01 - Copy retail_db into HDFS

Copy all the directories from retail_db under /tmp to HDFS (user HDFS home directory - /user/your_lab_id)

  • Hint: Use copyFromLocal or put to copy the files

  • Provide the output for below

    • Please list the files you have copied (using ls command)
    • Also get the results about files, blocks and locations on all the files that are copied.
2 Likes

drwxr-xr-x - tpulimoottil hdfs 0 2016-12-14 00:28 /user/tpulimoottil/retail_db/categories
drwxr-xr-x - tpulimoottil hdfs 0 2016-12-14 00:28 /user/tpulimoottil/retail_db/customers
drwxr-xr-x - tpulimoottil hdfs 0 2016-12-14 00:28 /user/tpulimoottil/retail_db/departments
drwxr-xr-x - tpulimoottil hdfs 0 2016-12-14 00:28 /user/tpulimoottil/retail_db/order_items
drwxr-xr-x - tpulimoottil hdfs 0 2016-12-14 00:28 /user/tpulimoottil/retail_db/orders
drwxr-xr-x - tpulimoottil hdfs 0 2016-12-14 00:28 /user/tpulimoottil/retail_db/products

[sumanthsharma21@gw01 ~]$ hadoop fs -copyFromLocal /tmp/retail_db/ /user/sumanth sharma21/
[sumanthsharma21@gw01 ~]$ hadoop fs -ls /user/sumanthsharma21/retail_db
Found 6 items
drwxr-xr-x - sumanthsharma21 hdfs 0 2016-12-14 00:32 /user/sumanthsharma21/retail_db/categories
drwxr-xr-x - sumanthsharma21 hdfs 0 2016-12-14 00:32 /user/sumanthsharma21/retail_db/customers
drwxr-xr-x - sumanthsharma21 hdfs 0 2016-12-14 00:32 /user/sumanthsharma21/retail_db/departments
drwxr-xr-x - sumanthsharma21 hdfs 0 2016-12-14 00:32 /user/sumanthsharma21/retail_db/order_items
drwxr-xr-x - sumanthsharma21 hdfs 0 2016-12-14 00:32 /user/sumanthsharma21/retail_db/orders
drwxr-xr-x - sumanthsharma21 hdfs 0 2016-12-14 00:32 /user/sumanthsharma21/retail_db/products
[sumanthsharma21@gw01 ~]$ hdfs fsck /user/sumanthsharma21/retail_db -files
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=sumanthsharma21&files=1&path=%2Fuser%2Fsumanthsharma21%2Fretail_db
FSCK started by sumanthsharma21 (auth:SIMPLE) from /172.16.1.100 for path /user/sumanthsharma21/retail_db at Wed Dec 14 00:33:28 EST 2016
/user/sumanthsharma21/retail_db
/user/sumanthsharma21/retail_db/categories
/user/sumanthsharma21/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
/user/sumanthsharma21/retail_db/customers
/user/sumanthsharma21/retail_db/customers/part-00000 953719 bytes, 1 block(s): OK
/user/sumanthsharma21/retail_db/departments
/user/sumanthsharma21/retail_db/departments/part-00000 60 bytes, 1 block(s): OK
/user/sumanthsharma21/retail_db/order_items
/user/sumanthsharma21/retail_db/order_items/part-00000 5408880 bytes, 1 block(s): OK
/user/sumanthsharma21/retail_db/orders
/user/sumanthsharma21/retail_db/orders/part-00000 2999944 bytes, 1 block(s): OK
/user/sumanthsharma21/retail_db/products
/user/sumanthsharma21/retail_db/products/part-00000 174155 bytes, 1 block(s): OK
Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:33:28 EST 2016 in 1 milliseconds

The filesystem under path ‘/user/sumanthsharma21/retail_db’ is HEALTHY

drwxr-xr-x - chethan121212 hdfs 0 2016-12-14 00:33 data/retail_db/categories
drwxr-xr-x - chethan121212 hdfs 0 2016-12-14 00:33 data/retail_db/customers
drwxr-xr-x - chethan121212 hdfs 0 2016-12-14 00:33 data/retail_db/departments
drwxr-xr-x - chethan121212 hdfs 0 2016-12-14 00:33 data/retail_db/order_items
drwxr-xr-x - chethan121212 hdfs 0 2016-12-14 00:33 data/retail_db/orders
drwxr-xr-x - chethan121212 hdfs 0 2016-12-14 00:33 data/retail_db/products

Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=chethan121212&files=1&path=%2Fuser%2Fchethan121212%2Fdata%2Fretail_db
FSCK started by chethan121212 (auth:SIMPLE) from /172.16.1.100 for path /user/chethan121212/data/retail_db at Wed Dec 14 00:41:09 EST 2016
/user/chethan121212/data/retail_db
/user/chethan121212/data/retail_db/categories
/user/chethan121212/data/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
/user/chethan121212/data/retail_db/customers
/user/chethan121212/data/retail_db/customers/part-00000 953719 bytes, 1 block(s): OK
/user/chethan121212/data/retail_db/departments
/user/chethan121212/data/retail_db/departments/part-00000 60 bytes, 1 block(s): OK
/user/chethan121212/data/retail_db/order_items
/user/chethan121212/data/retail_db/order_items/part-00000 5408880 bytes, 1 block(s): OK
/user/chethan121212/data/retail_db/orders
/user/chethan121212/data/retail_db/orders/part-00000 2999944 bytes, 1 block(s): OK
/user/chethan121212/data/retail_db/products
/user/chethan121212/data/retail_db/products/part-00000 174155 bytes, 1 block(s): OK
Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:41:09 EST 2016 in 1 milliseconds

The filesystem under path ‘/user/chethan121212/data/retail_db’ is HEALTHY

Copying to HDFS:
hadoop fs -ls /user/jasonbourne/retail_db
Found 6 items
drwxr-xr-x - jasonbourne jasonbourne 0 2016-12-14 00:30 /user/jasonbourne/retail_db/categories
drwxr-xr-x - jasonbourne jasonbourne 0 2016-12-14 00:30 /user/jasonbourne/retail_db/customers
drwxr-xr-x - jasonbourne jasonbourne 0 2016-12-14 00:30 /user/jasonbourne/retail_db/departments
drwxr-xr-x - jasonbourne jasonbourne 0 2016-12-14 00:30 /user/jasonbourne/retail_db/order_items
drwxr-xr-x - jasonbourne jasonbourne 0 2016-12-14 00:30 /user/jasonbourne/retail_db/orders
drwxr-xr-x - jasonbourne jasonbourne 0 2016-12-14 00:30 /user/jasonbourne/retail_db/products

File Blocks and Locations
hdfs fsck /user/jasonbourne/retail_db -files -blocks -locations
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=jasonbourne&files=1&blocks=1&locations=1&path=%2Fuser%2Fjasonbourne%2Fretail_db
FSCK started by jasonbourne (auth:SIMPLE) from /172.16.1.100 for path /user/jasonbourne/retail_db at Wed Dec 14 00:35:12 EST 2016
/user/jasonbourne/retail_db
/user/jasonbourne/retail_db/categories
/user/jasonbourne/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867265_126881 len=1029 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK]]

Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1

Copying to HDFS
hdfs dfs -put /tmp/retail_db /user/nagellarajashyam/

Listing:
[nagellarajashyam@gw01 ~]$ hdfs dfs -ls /user/nagellarajashyam/retail_db
Found 6 items
drwxr-xr-x - nagellarajashyam hdfs 0 2016-12-14 00:30 /user/nagellarajashyam/retail_db/categories
drwxr-xr-x - nagellarajashyam hdfs 0 2016-12-14 00:30 /user/nagellarajashyam/retail_db/customers
drwxr-xr-x - nagellarajashyam hdfs 0 2016-12-14 00:30 /user/nagellarajashyam/retail_db/departments
drwxr-xr-x - nagellarajashyam hdfs 0 2016-12-14 00:30 /user/nagellarajashyam/retail_db/order_items
drwxr-xr-x - nagellarajashyam hdfs 0 2016-12-14 00:30 /user/nagellarajashyam/retail_db/orders
drwxr-xr-x - nagellarajashyam hdfs 0 2016-12-14 00:30 /user/nagellarajashyam/retail_db/products

file system checking:
[nagellarajashyam@gw01 ~]$ hdfs fsck /user/nagellarajashyam/retail_db -files -blocks -locations
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=nagellarajashyam&files=1&blocks=1&locations=1&path=%2Fuser%2Fnagellarajashyam%2Fretail_db
FSCK started by nagellarajashyam (auth:SIMPLE) from /172.16.1.100 for path /user/nagellarajashyam/retail_db at Wed Dec 14 00:37:36 EST 2016
/user/nagellarajashyam/retail_db
/user/nagellarajashyam/retail_db/categories
/user/nagellarajashyam/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867262_126878 len=1029 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK]]

/user/nagellarajashyam/retail_db/customers
/user/nagellarajashyam/retail_db/customers/part-00000 953719 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867263_126879 len=953719 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK]]

/user/nagellarajashyam/retail_db/departments
/user/nagellarajashyam/retail_db/departments/part-00000 60 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867264_126880 len=60 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK]]

/user/nagellarajashyam/retail_db/order_items
/user/nagellarajashyam/retail_db/order_items/part-00000 5408880 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867266_126882 len=5408880 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK]]

/user/nagellarajashyam/retail_db/orders
/user/nagellarajashyam/retail_db/orders/part-00000 2999944 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867267_126883 len=2999944 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK]]

/user/nagellarajashyam/retail_db/products
/user/nagellarajashyam/retail_db/products/part-00000 174155 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867269_126885 len=174155 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK]]

Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:37:36 EST 2016 in 1 milliseconds

The filesystem under path ‘/user/nagellarajashyam/retail_db’ is HEALTHY

[mahesh007@gw01 ~]$ hadoop fs -copyFromLocal /tmp/retail_db /user/mahesh007
[mahesh007@gw01 ~]$ hadoop fs -ls /user/mahesh007/retail_db
Found 6 items
drwxr-xr-x - mahesh007 hdfs 0 2016-12-14 00:36 /user/mahesh007/retail_db/categori es
drwxr-xr-x - mahesh007 hdfs 0 2016-12-14 00:36 /user/mahesh007/retail_db/customer s
drwxr-xr-x - mahesh007 hdfs 0 2016-12-14 00:36 /user/mahesh007/retail_db/departme nts
drwxr-xr-x - mahesh007 hdfs 0 2016-12-14 00:36 /user/mahesh007/retail_db/order_it ems
drwxr-xr-x - mahesh007 hdfs 0 2016-12-14 00:36 /user/mahesh007/retail_db/orders
drwxr-xr-x - mahesh007 hdfs 0 2016-12-14 00:36 /user/mahesh007/retail_db/products

[mahesh007@gw01 ~]$ hdfs fsck /user/mahesh007/retail_db -files -blocks -locations
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=mahesh007&files=1&blocks=1&locations=1&path=%2Fuser%2Fmahesh007%2Fretail_db
FSCK started by mahesh007 (auth:SIMPLE) from /172.16.1.100 for path /user/mahesh007/retail_db at Wed Dec 14 00:47:04 EST 2016
/user/mahesh007/retail_db
/user/mahesh007/retail_db/categories
/user/mahesh007/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867321_126937 len=1029 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK]]

/user/mahesh007/retail_db/customers
/user/mahesh007/retail_db/customers/part-00000 953719 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867322_126938 len=953719 repl=3 [DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK]]

/user/mahesh007/retail_db/departments
/user/mahesh007/retail_db/departments/part-00000 60 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867323_126939 len=60 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK]]

/user/mahesh007/retail_db/order_items
/user/mahesh007/retail_db/order_items/part-00000 5408880 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867324_126940 len=5408880 repl=3 [DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK]]

/user/mahesh007/retail_db/orders
/user/mahesh007/retail_db/orders/part-00000 2999944 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867325_126941 len=2999944 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK]]

/user/mahesh007/retail_db/products
/user/mahesh007/retail_db/products/part-00000 174155 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867326_126942 len=174155 repl=3 [DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK]]

Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:47:04 EST 2016 in 0 milliseconds

The filesystem under path ‘/user/mahesh007/retail_db’ is HEALTHY

drwxr-xr-x - aruncse11 hdfs 0 2016-12-14 00:27 /user/aruncse11/arun/retail_db/categories
drwxr-xr-x - aruncse11 hdfs 0 2016-12-14 00:27 /user/aruncse11/arun/retail_db/customers
drwxr-xr-x - aruncse11 hdfs 0 2016-12-14 00:27 /user/aruncse11/arun/retail_db/departments
drwxr-xr-x - aruncse11 hdfs 0 2016-12-14 00:27 /user/aruncse11/arun/retail_db/order_items
drwxr-xr-x - aruncse11 hdfs 0 2016-12-14 00:27 /user/aruncse11/arun/retail_db/orders
drwxr-xr-x - aruncse11 hdfs 0 2016-12-14 00:27 /user/aruncse11/arun/retail_db/products

/user/aruncse11/arun/retail_db
/user/aruncse11/arun/retail_db/categories
/user/aruncse11/arun/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
/user/aruncse11/arun/retail_db/customers
/user/aruncse11/arun/retail_db/customers/part-00000 953719 bytes, 1 block(s): OK
/user/aruncse11/arun/retail_db/departments
/user/aruncse11/arun/retail_db/departments/part-00000 60 bytes, 1 block(s): OK
/user/aruncse11/arun/retail_db/order_items
/user/aruncse11/arun/retail_db/order_items/part-00000 5408880 bytes, 1 block(s): OK
/user/aruncse11/arun/retail_db/orders
/user/aruncse11/arun/retail_db/orders/part-00000 2999944 bytes, 1 block(s): OK
/user/aruncse11/arun/retail_db/products
/user/aruncse11/arun/retail_db/products/part-00000 174155 bytes, 1 block(s): OK
Status: HEALTHY

hdfs dfs -ls /user/catchask/retail_db
Found 6 items
drwxr-xr-x - catchask hdfs 0 2016-12-14 00:30 /user/catchask/retail_db/categories
drwxr-xr-x - catchask hdfs 0 2016-12-14 00:30 /user/catchask/retail_db/customers
drwxr-xr-x - catchask hdfs 0 2016-12-14 00:30 /user/catchask/retail_db/departments
drwxr-xr-x - catchask hdfs 0 2016-12-14 00:30 /user/catchask/retail_db/order_items
drwxr-xr-x - catchask hdfs 0 2016-12-14 00:30 /user/catchask/retail_db/orders
drwxr-xr-x - catchask hdfs 0 2016-12-14 00:30 /user/catchask/retail_db/products

hdfs fsck /user/catchask/retail_db/categories
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=catchask&path=%2Fuser%2Fcatchask%2Fretail_db%2Fcategories
FSCK started by catchask (auth:SIMPLE) from /172.16.1.100 for path /user/catchask/retail_db/categories at Wed Dec 14 00:39:49 EST 2016
.Status: HEALTHY
Total size: 1029 B
Total dirs: 1
Total files: 1
Total symlinks: 0
Total blocks (validated): 1 (avg. block size 1029 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:39:49 EST 2016 in 0 milliseconds

Found 6 items
drwxr-xr-x - saswat232 hdfs 0 2016-12-14 00:31 /user/saswat232/retail_db/categories
drwxr-xr-x - saswat232 hdfs 0 2016-12-14 00:31 /user/saswat232/retail_db/customers
drwxr-xr-x - saswat232 hdfs 0 2016-12-14 00:31 /user/saswat232/retail_db/departments
drwxr-xr-x - saswat232 hdfs 0 2016-12-14 00:31 /user/saswat232/retail_db/order_items
drwxr-xr-x - saswat232 hdfs 0 2016-12-14 00:31 /user/saswat232/retail_db/orders
drwxr-xr-x - saswat232 hdfs 0 2016-12-14 00:31 /user/saswat232/retail_db/products

FSCK::

…Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:35:32 EST 2016 in 1 milliseconds

$ hadoop fs -put retail_db
$ hdfs dfs -lsr retail_db
lsr: DEPRECATED: Please use ‘ls -R’ instead.
drwxr-xr-x - infosnehasish hdfs 0 2016-12-14 00:26 retail_db/categories
-rw-r–r-- 3 infosnehasish hdfs 1029 2016-12-14 00:26 retail_db/categories/part-00000
drwxr-xr-x - infosnehasish hdfs 0 2016-12-14 00:26 retail_db/customers
-rw-r–r-- 3 infosnehasish hdfs 953719 2016-12-14 00:26 retail_db/customers/part-00000
drwxr-xr-x - infosnehasish hdfs 0 2016-12-14 00:26 retail_db/departments
-rw-r–r-- 3 infosnehasish hdfs 60 2016-12-14 00:26 retail_db/departments/part-00000
drwxr-xr-x - infosnehasish hdfs 0 2016-12-14 00:26 retail_db/order_items
-rw-r–r-- 3 infosnehasish hdfs 5408880 2016-12-14 00:26 retail_db/order_items/part-00000
drwxr-xr-x - infosnehasish hdfs 0 2016-12-14 00:26 retail_db/orders
-rw-r–r-- 3 infosnehasish hdfs 2999944 2016-12-14 00:26 retail_db/orders/part-00000
drwxr-xr-x - infosnehasish hdfs 0 2016-12-14 00:26 retail_db/products
-rw-r–r-- 3 infosnehasish hdfs 174155 2016-12-14 00:26 retail_db/products/part-00000


hdfs fsck /user/infosnehasish/retail_db -files -blocks -locations

[infosnehasish@gw01 tmp]$ hdfs fsck /user/infosnehasish/retail_db -files -blocks -locations
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=infosnehasish&files=1&blocks=1&locations=1&path=%2Fuser%2Finfosnehasish%2Fretail_db
FSCK started by infosnehasish (auth:SIMPLE) from /172.16.1.100 for path /user/infosnehasish/retail_db at Wed Dec 14 00:37:04 EST 2016
/user/infosnehasish/retail_db
/user/infosnehasish/retail_db/categories
/user/infosnehasish/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867199_126813 len=1029 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK]]

/user/infosnehasish/retail_db/customers
/user/infosnehasish/retail_db/customers/part-00000 953719 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867200_126814 len=953719 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK]]

/user/infosnehasish/retail_db/departments
/user/infosnehasish/retail_db/departments/part-00000 60 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867201_126815 len=60 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK]]

/user/infosnehasish/retail_db/order_items
/user/infosnehasish/retail_db/order_items/part-00000 5408880 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867202_126816 len=5408880 repl=3 [DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK]]

/user/infosnehasish/retail_db/orders
/user/infosnehasish/retail_db/orders/part-00000 2999944 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867203_126817 len=2999944 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK]]

/user/infosnehasish/retail_db/products
/user/infosnehasish/retail_db/products/part-00000 174155 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867204_126818 len=174155 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK]]

Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:37:04 EST 2016 in 1 milliseconds

[sumanthsharma21@gw01 ~]$ hdfs fsck /user/sumanthsharma21/retail_db -files -blocks -locations
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=sumanthsharma21&files=1&blocks=1&locations=1&path=%2Fuser%2Fsumanthsharma21%2Fretail_db
FSCK started by sumanthsharma21 (auth:SIMPLE) from /172.16.1.100 for path /user/sumanthsharma21/retail_db at Wed Dec 14 00:44:05 EST 2016
/user/sumanthsharma21/retail_db
/user/sumanthsharma21/retail_db/categories
/user/sumanthsharma21/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867288_126904 len=1029 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK]]

/user/sumanthsharma21/retail_db/customers
/user/sumanthsharma21/retail_db/customers/part-00000 953719 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867289_126905 len=953719 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK]]

/user/sumanthsharma21/retail_db/departments
/user/sumanthsharma21/retail_db/departments/part-00000 60 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867290_126906 len=60 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK]]

/user/sumanthsharma21/retail_db/order_items
/user/sumanthsharma21/retail_db/order_items/part-00000 5408880 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867291_126907 len=5408880 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK]]

/user/sumanthsharma21/retail_db/orders
/user/sumanthsharma21/retail_db/orders/part-00000 2999944 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867292_126908 len=2999944 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK]]

/user/sumanthsharma21/retail_db/products
/user/sumanthsharma21/retail_db/products/part-00000 174155 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867293_126909 len=174155 repl=3 [DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK]]

Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:44:05 EST 2016 in 1 milliseconds

The filesystem under path ‘/user/sumanthsharma21/retail_db’ is HEALTHY

hdfs dfs -put /tmp/retail_db/ /user/parulshine92/

hdfs dfs -ls /user/parulshine92/retail_db
Found 6 items
drwxr-xr-x - parulshine92 hdfs 0 2016-12-14 00:28 /user/parulshine92/retail_db/categories
drwxr-xr-x - parulshine92 hdfs 0 2016-12-14 00:28 /user/parulshine92/retail_db/customers
drwxr-xr-x - parulshine92 hdfs 0 2016-12-14 00:28 /user/parulshine92/retail_db/departments
drwxr-xr-x - parulshine92 hdfs 0 2016-12-14 00:28 /user/parulshine92/retail_db/order_items
drwxr-xr-x - parulshine92 hdfs 0 2016-12-14 00:28 /user/parulshine92/retail_db/orders
drwxr-xr-x - parulshine92 hdfs 0 2016-12-14 00:28 /user/parulshine92/retail_db/products
[parulshine92@gw01 retail_db]$ hdfs dfs -ls /user/parulshine92/retail_db/categories

hdfs fsck /user/parulshine92/retail_db -files -blocks -locations
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=parulshine92&files=1&blocks=1&locations=1&path=%2Fuser%2Fparulshine92%2Fretail_db
FSCK started by parulshine92 (auth:SIMPLE) from /172.16.1.100 for path /user/parulshine92/retail_db at Wed Dec 14 00:37:28 EST 2016
/user/parulshine92/retail_db
/user/parulshine92/retail_db/categories
/user/parulshine92/retail_db/categories/part-00000 1029 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867245_126861 len=1029 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK]]

/user/parulshine92/retail_db/customers
/user/parulshine92/retail_db/customers/part-00000 953719 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867246_126862 len=953719 repl=3 [DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK]]

/user/parulshine92/retail_db/departments
/user/parulshine92/retail_db/departments/part-00000 60 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867247_126863 len=60 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK]]

/user/parulshine92/retail_db/order_items
/user/parulshine92/retail_db/order_items/part-00000 5408880 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867248_126864 len=5408880 repl=3 [DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK]]

/user/parulshine92/retail_db/orders
/user/parulshine92/retail_db/orders/part-00000 2999944 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867249_126865 len=2999944 repl=3 [DatanodeInfoWithStorage[172.16.1.104:50010,DS-98fec5a6-72a9-4590-99cc-cee3a51f4dd5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-b0f1636e-fd08-4ddb-bba9-9df8868dfb5d,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-1f4edfab-2926-45f9-a37c-ae9d1f542680,DISK]]

/user/parulshine92/retail_db/products
/user/parulshine92/retail_db/products/part-00000 174155 bytes, 1 block(s): OK
0. BP-292116404-172.16.1.101-1479167821718:blk_1073867250_126866 len=174155 repl=3 [DatanodeInfoWithStorage[172.16.1.104:50010,DS-f4667aac-0f2c-463c-9584-d625928b9af5,DISK], DatanodeInfoWithStorage[172.16.1.102:50010,DS-1edb1d35-81bf-471b-be04-11d973e2a832,DISK], DatanodeInfoWithStorage[172.16.1.103:50010,DS-7fb58858-abe9-4a52-9b75-755d849a897b,DISK]]

Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:37:28 EST 2016 in 1 milliseconds

The filesystem under path ‘/user/parulshine92/retail_db’ is HEALTHY

[farhanmisarwala@gw01 data]$ hadoop fs -ls /user/farhanmisarwala/data/retail_db
Found 6 items
drwxr-xr-x - farhanmisarwala hdfs 0 2016-12-14 00:32 /user/farhanmisarwala/data/retail_db/categories
drwxr-xr-x - farhanmisarwala hdfs 0 2016-12-14 00:32 /user/farhanmisarwala/data/retail_db/customers
drwxr-xr-x - farhanmisarwala hdfs 0 2016-12-14 00:32 /user/farhanmisarwala/data/retail_db/departments
drwxr-xr-x - farhanmisarwala hdfs 0 2016-12-14 00:32 /user/farhanmisarwala/data/retail_db/order_items
drwxr-xr-x - farhanmisarwala hdfs 0 2016-12-14 00:32 /user/farhanmisarwala/data/retail_db/orders
drwxr-xr-x - farhanmisarwala hdfs 0 2016-12-14 00:32 /user/farhanmisarwala/data/retail_db/products

[farhanmisarwala@gw01 data]$ hdfs fsck /user/farhanmisarwala/data/retail_db
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=farhanmisarwala&path=%2Fuser%2Ffarhanmisarwala%2Fdata%2Fretail_db
FSCK started by farhanmisarwala (auth:SIMPLE) from /172.16.1.100 for path /user/farhanmisarwala/data/retail_db at Wed Dec 14 00:43:46 EST 2016
…Status: HEALTHY
Total size: 9537787 B
Total dirs: 7
Total files: 6
Total symlinks: 0
Total blocks (validated): 6 (avg. block size 1589631 B)
Minimally replicated blocks: 6 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:43:46 EST 2016 in 0 milliseconds

The filesystem under path ‘/user/farhanmisarwala/data/retail_db’ is HEALTHY

[farhanmisarwala@gw01 data]$ hdfs fsck /user/farhanmisarwala/data/retail_db/customers
Connecting to namenode via http://nn01.itversity.com:50070/fsck?ugi=farhanmisarwala&path=%2Fuser%2Ffarhanmisarwala%2Fdata%2Fretail_db%2Fcustomers
FSCK started by farhanmisarwala (auth:SIMPLE) from /172.16.1.100 for path /user/farhanmisarwala/data/retail_db/customers at Wed Dec 14 00:45:21 EST 2016
.Status: HEALTHY
Total size: 953719 B
Total dirs: 1
Total files: 1
Total symlinks: 0
Total blocks (validated): 1 (avg. block size 953719 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Dec 14 00:45:21 EST 2016 in 0 milliseconds

The filesystem under path ‘/user/farhanmisarwala/data/retail_db/customers’ is HEALTHY