Input file doesn't exist for spark application in Intellij Idea

apache-spark
scala

#1

Hi, I’ve configured winutils.exe to support spark in windows machine. While I’m trying to develop sample spark application in Intellij Idea, I was repeatedly getting “input file path doesn’t exist”. Since, i had a file at local, hdfs . File name is “one.txt”
I know that loading a local file syntax is : scala.io.Source.fromFile(“filepath”)
for hdfs location means: sc.textFile(“filepath”)

In Intellij idea I’ve used to load file as:
//file loading from hdfs location
val data = sc.textFile(“hdfs://nn01.itversity.com:8020/user/saikiranchepuri35/one.txt”)
// file loading from local file location
// val data1 = Source.fromFile("/home/saikiranchepuri35/one.txt").getLines().toList
// val data = sc.parallelize(data1)

val rdd1 = data.flatMap(_.split(",")).map(word=>(word,1))

rdd1.collect.foreach(println)

Can anyone please help me out how to give exactly the correct path to work on it. As I’m facing this issue from morning to evening and went the videos couldn’t resolve it.

Exception in thread “main” java.io.FileNotFoundException: \home\saikiranchepuri35\one.txt (The system cannot find the path specified)
at java.io.FileInputStream.open0(Native Method)


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster


#2

Please go through this topic