Running Spark applications using Scala and Python on EMR Cluster : ERROR


#1

Hello Itversity-

i am following the below video.

At time 1:03:28, Durga explains about port forwarding using SSH. However I am using putty to connect from window. I am facing some issue. If someone can help me on it it will be great.

Steps I followed:

Created a spark EMR cluster using cluster mode.

Then established a connection to gatewaynode/master node thru putty using the steps suggested under SSH in cluster summary page.

Then I was trying to use port forwarding to connect to other node using SSH. But getting the below error.

EEEEEEEEEEEEEEEEEEEE MMMMMMMM MMMMMMMM RRRRRRRRRRRRRRR
E::::::::::::::::::E M:::::::M M:::::::M R::::::::::::::R
EE:::::EEEEEEEEE:::E M::::::::M M::::::::M R:::::RRRRRR:::::R
E::::E EEEEE M:::::::::M M:::::::::M RR::::R R::::R
E::::E M::::::m:::M M:::m::::::M R:::R R::::R
E:::::EEEEEEEEEE M:::::M M:::M M:::M M:::::M R:::RRRRRR:::::R
E::::::::::::::E M:::::M M:::m:::M M:::::M R:::::::::::RR
E:::::EEEEEEEEEE M:::::M M:::::M M:::::M R:::RRRRRR::::R
E::::E M:::::M M:::M M:::::M R:::R R::::R
E::::E EEEEE M:::::M MMM M:::::M R:::R R::::R
EE:::::EEEEEEEE::::E M:::::M M:::::M R:::R R::::R
E::::::::::::::::::E M:::::M M:::::M RR::::R R::::R
EEEEEEEEEEEEEEEEEEEE MMMMMMM MMMMMMM RRRRRRR RRRRRR

[hadoop@ip-172-31-29-40 ~]$ ssh hadoop@ip-172-31-28-51.ap-south-1.compute.intern al
The authenticity of host ‘ip-172-31-28-51.ap-south-1.compute.internal (172.31.28 .51)’ can’t be established.
ECDSA key fingerprint is SHA256:+bCUkh99NhRRms9pPU3uaDDjN/W9NTQCnsY8tBeOQLc.
ECDSA key fingerprint is MD5:00:0d:aa:20:03:83:01:98:86:18:6d:39:38:00:ce:d6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘ip-172-31-28-51.ap-south-1.compute.internal,172.31.2 8.51’ (ECDSA) to the list of known hosts.
Permission denied (publickey).

Regards,
Jeevan


#2

Hello Itversity-

I resolved this issue with cygwin.

However I followed some steps to connect from cygwin.

$ ssh-add -K ~/.ssh/bdclouddemo.pem
Could not open a connection to your authentication agent.

I did below steps.

eval “$(ssh-agent)”
ssh-agent /bin/sh

$ ssh-add -k ~/.ssh/bdclouddemo.pem
Identity added: /home/jeevandas/.ssh/bdclouddemo.pem (/home/jeevandas/.ssh/bdclouddemo.pem)

Question -

Can anyone explain me what the below steps are doing ?

eval “$(ssh-agent)”
ssh-agent /bin/sh

Regards,
Jeevan


#3