How to access hdfs file in Intellij Idea that installed on window machine

apache-spark
hdfs
scala

#1

Hi,
I’m able to access the textfile at spark-shell thru
val words = sc.textFile("/public/randomtextwriter/part-m-00000") and I’ve done transformations, actions, finally submitted the job. So it is running fine at “spark-shell”.

But when I try to access the same file at IntelliJ Idea, I’m getting an error saying:
Exception in thread “main” org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/public/randomtextwriter/part-m-00000
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)

Since, I’ve installed IntelliJ idea on windows machine. I’m facing issue only with when I try to access the hdfs file but not with the local file sys.
syntax:

val wc = sc.textFile("/public/randomtextwriter/part-m-00000") // error: input path not found.
val rawdata = Source.fromFile(“C:\Users\Sony\Desktop\hadoop\datasets\wordcount.txt”).getLines().toList // working fine, thru local file sys.

Please please help me on this.

Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster