Could not find the input file path in scala using cygwin

Hi,
could not find the input file path while reading text file from cygwin

Here is the code am running
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
val conf = new SparkConf().setAppName(“Testing”).setMaster(“local”)
val sc = new SparkContext(conf)
val randomtext = sc.textFile(“file:///cygdrive/c/Users/govtham/wordcount.txt”)
val fm = randomtext.flatMap(rec => rec.split(" "))
val m = fm.map((_, 1))
val r = m.reduceByKey( _ + _ )

Can you please suggest

Thanks
Goutham

@Goutham

I understand you are using file:/// option to get the file from local. But here local means linux local /home/user , but it seems you are referring to windows local c drive

thanks for reply
i tried but not working

Please see the code below:
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
val conf = new SparkConf().setAppName(“Testing”).setMaster(“local”)
val sc = new SparkContext(conf)
val randomtext = sc.textFile("/home/govtham/Research/data/wordcount.txt")
val fm = randomtext.flatMap(rec => rec.split(" "))
val m = fm.map((_, 1))
val r = m.reduceByKey( _ + _ )

Thanks
Goutham

@Goutham - Check you have data in below location

cd /home/govtham/Research/data/
ls -ltr

Here is the data in the file

Thanks
Goutham

@Goutham - It’s look like you are using your own VM not big data labs.

home/govtham/Research/data/ - Refers to big data labs local

can you give pwd and repalce with file path accordingly.

Thanks Gnan for reply,
actually, i was trying in my window machine using cygwin, i did not install hadoop, i think we need hadoop right…?
I have setup the sbt and IntelliJ in my window machine and what am doing is…
sbt console
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
val conf = new SparkConf().setAppName(“Testing”).setMaster(“local”)
val sc = new SparkContext(conf)
val randomtext = sc.textFile("/home/govtham/Research/data/wordcount.txt")
randomtext.collect().foreach(println)

Now i tried in big data labs, i could able to read
val cate = sc.textFile("/user/govtham9/sqoop_import/categories")
cate.take(5).foreach(println)

Please tell me one, if i want to read the file in my window machine using cygwin how can i achive that?

Thanks
Goutham

@Goutham -

I haven’t tried in windows, In MAC sc.texFile("/Users/…") will work.

you can try sc.textFile("/cygdrive/c/…")

Otherwise you can push the file from your desktop to gateway node using SCP command
scp @gw01.itversity.com:~

Goutham,
After spending 3 hours finally I fixed the same problem as you are facing… use the below format and you should be able to load the file in your local machine using Cygwin

val orders = sc.textFile(“file:/C:/Users/Rams/IdeaProjects/POC/src/rams.txt”)

1 Like