Introduction to Hadoop eco system - Overview of HDFS - Managing HDFS Directories

In this article, we will learn how to create directories and manage ownership in HDFS.

Key Concept 1

By default, hdfs is the superuser of HDFS. Use hadoop fs -mkdir or hdfs dfs -mkdir to create directories. Change ownership of files using hadoop fs -chown or hdfs dfs -chown. You can also change the group using the -chgrp command.

Key Concept 2

To create user space in HDFS:

  1. Create a directory with the user ID itversity under /user.
  2. Change ownership to the same name as the directory created earlier (/user/itversity).
  3. Validate permissions using hadoop fs -ls or hdfs dfs -ls on /user.

Hands-On Tasks

  1. Create a user space in HDFS for itversity by running the following commands:
sudo -u hdfs hdfs dfs -mkdir /user/itversity
sudo -u hdfs hdfs dfs -chown -R itversity:students /user/itversity
hdfs dfs -ls /user|grep itversity
  1. Create folders under your home directory by running:
hdfs dfs -ls /user/${USER}
hdfs dfs -mkdir /user/${USER}/retail_db
hdfs dfs -ls /user/${USER}
  1. Create a directory structure using mkdir -p:
hdfs dfs -mkdir -p /user/${USER}/retail_db/orders/year=2020
hdfs dfs -ls -R /user/${USER}/retail_db
  1. Delete directories using hdfs dfs -rm -R for non-empty directories and hdfs dfs -rmdir for empty directories.
hdfs dfs -rmdir /user/${USER}/retail_db/orders/year=2020
hdfs dfs -rmdir /user/${USER}/retail_db
hdfs dfs -rm -R /user/${USER}/retail_db
hdfs dfs -ls /user/${USER}

Keep practicing these tasks to strengthen your understanding.

Conclusion

In this article, we covered the key concepts of managing HDFS directories, including creating, changing ownership, and deleting directories. We encourage you to practice these tasks and engage with the community for further learning.

Watch the video tutorial here