HDFS linking to spark/hive

apache-spark

#1

ISSUE WITH HIVE TO HDFS == >
hive (ashoo1234)> create external table movie_ratings1 (id string, id1 string, id2 string,id3 string) row format delimited fields terminated by ’ ’ location ‘/user/ashoo1234/moviesdata/u.data’
> ;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:hdfs://nn01.itversity.com:8020/user/ashoo1234/moviesdata/u.data is not a directory or unable to create one)

Here is the file on HDFS == >
[ashoo1234@gw03 ~]$ hdfs dfs -ls /user/ashoo1234/moviesdata/u.data
-rw-r–r-- 3 ashoo1234 hdfs 1979173 2018-01-27 13:18 /user/ashoo1234/moviesdata/u.data

Since the URL pointed to “nn01” , I tried ssh. but that too failed. “nn01-itversity-com” . Permission denied.


#2

[ashoo1234@gw03 ~]$ ssh nn01.itversity.com
The authenticity of host ‘nn01.itversity.com (172.16.1.101)’ can’t be established.
RSA key fingerprint is SHA256:QBSGJKz+3vX3MszPerRobakUTAUAR9o/NVKZi8COUqE.
RSA key fingerprint is MD5:86:5c:f8:32:a7:0c:80:14:e6:d7:de:e8:44:78:42:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘nn01.itversity.com,172.16.1.101’ (RSA) to the list of known hosts.
ashoo1234@nn01.itversity.com’s password:
Permission denied, please try again.


#3

Please try the below command. It is working fine.

create external table movie_ratings1 (id string, id1 string, id2 string,id3 string) row format delimited fields terminated by '' location '/user/ashoo1234/moviesdata/'

You won’t be able to connect to nn01.