Spark with cygwin installation failure with loading main class failure


#1

Hi ,
I am trying to integrate spark with cygwin by following the steps as per the video which blocked my installation with the following issue .

PFA for details .


#2

@Sreedhar_Paladugu
Try running spark-shell.cmd

And a document has shared for all workshop participants in Whatsapp group. Please try to troubleshoot with that if any steps are missing.


#3

I’m facing the same error too. I have spent quite some time debugging. My java environment variables have been set up and I have also set up Spark environment variables to .bash_profile and .profile.

I’m running Cygwin as administrator but unable to execute chmod +x spark­shell.cmd

Here are the errors I’m getting:
$ spark-shell
Error: Could not find or load main class org.apache.spark.launcher.Main

$ spark-shell.cmd
-bash: /home/mc58838/spark/bin/spark-shell.cmd: Permission denied


#4

Use

chmod +x /home/mc58838/spark/bin/spark-shell.cmd

And then run again


#5

I was able to execute chmod +x /home/mc58838/spark/bin/spark-shell.cmd

Now the error is:

$ spark-shell.cmd
The system cannot find the path specified.

I tried to execute it from the home directory as well as going to the bin directory where the .cmd file is located.
Plase note I had also created a soft link using ‘ln’ command. I don’t know if that could be an issue.


#6

Hi vinod i followed your steps which triggered new error specified below.

$ spark-shell.cmd
’cmd’ is not recognized as an internal or external command,
operable program or batch file.


#7

Can you just check the .bash_profile file once.

And send me output for echo $SPARK_HOME?


#8

i cant find .bash_profile file .
But when i did echo $SPARK_HOME it gave me /home/sreedhar/spark


#10

when i run env command its giving me the following which is invalid as my new path is

C:\cygwin64\home\vijayendra\spark-1.6.3-bin-hadoop2.6

PATH=/usr/local/bin:/usr/bin:/cygdrive/c/Python27:/home/vijayendra/spark-1.6.3-bin-hadoop2.6/bin:/home/vijayendra/spark-1.6.3-bin-hadoop2.6/sbin:/cygdrive/c/hadoop/bin:/cygdrive/c/Users/vijayendra/AppData/Local/Microsoft/WindowsApps:/cygdrive/c/Program Files/Java/jre-9/bin:/cygdrive/c/Users/vijayendra/AppData/Local/atom/bin
PS1=[\e]0;\w\a]\n[\e[32m]\u@\h [\e[33m]\w[\e[0m]\n$
HOMEDRIVE=C:
INFOPATH=/usr/local/info:/usr/share/info:/usr/info
HOMEPATH=\Users\vijayendra
ORIGINAL_PATH=/cygdrive/c/Python27:/home/vijayendra/spark-1.6.3-bin-hadoop2.6/bin:/home/vijayendra/spark-1.6.3-bin-hadoop2.6/sbin:/cygdrive/c/hadoop/bin:/cygdrive/c/Users/vijayendra/AppData/Local/Microsoft/WindowsApps:/cygdrive/c/Program Files/Java/jre-9/bin:/cygdrive/c/Users/vijayendra/AppData/Local/atom/bin
EXECIGNORE=*.dll
VBOX_MSI_INSTALL_PATH=E:\virtualboxinstall
_=/usr/bin/env


#11

This is my temp as i am about to get my macbook in a week or so .
Please assist me in completing the setup so that i can practice for this machine or so .