Setting up bigdata lab in ITversity


#1

Hi Itversity Team and @dgadiraju ,

You have done a fabulous job by giving access to a good cluster which is helpful for the beginners like me.

Since I am crawling baby in Big data technology i need your assistance in setting big data labs in itversity.

I logged into my account as “pruthvibigdata”

Initially i ran hadoop fs -ls command , i got no results.

Looks like the serivices like hadoop, hive , and other folders are missing.

so to continue with my learning, I copied “retail_db” from /public folder to /user/pruthvibigdata folder

then I issued the command hive and hence i can see the output of hadoop fs -ls is as below.
Inline image 1

As per the instructions in video tutorials, i started creating external table on hive using below syntax
Query:
CREATE EXTERNAL TABLE PRUTHVIBIGDATA.PRODUCTS
(
PRODUCT_ID INT,
PRODUCT_CATEGORY_ID INT,
PRODUCT_NAME VARCHAR(45),
PRODUCT_DESCRIPTION VARCHAR(255),
PRODUCT_PRICE FLOAT,
PRODUCT_IMAGE VARCHAR(255)
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION ‘/USER/PRUTHVIBIGDATA/RETAIL_DB/PRODUCTS’;

and got the below error:

java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.security.AccessControlException: Permission denied: user=pruthvibigdata, access=WRITE, inode="/":hdfs:hdfs:drwxrwxr-x atorg.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1827) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1811) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1785) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAccess(FSNamesystem.java:8558) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkAccess(NameNodeRpcServer.java:2064) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.checkAccess(ClientNamenodeProtocolServerSideTranslatorPB.java:1451) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) )

Could you please help me in

  1. Setting up the labs
  2. Using the service on cluster
  3. Rectify my error

Thanks in advance.

Looking forward for some help.


#2

Please check the below screenshot to see how to use hadoop fs -ls command. It is working fine.