How to connect my local RDBMS to itversity lab for spark


#1

Hi,

I have mysql in my local machine and there is a data in it and I need to use that data in spark.

earlier I have connected itversity’s mysql in spark and it worked fine and I used the following code

spark-shell --jars /usr/share/java/mysql-connector-java.jar

val dataframe_mysql = sqlContext.read.format("jdbc")
.option("url", "jdbc:mysql://ms.itversity.com:3306/retail_db")
.option("driver", "com.mysql.jdbc.Driver")
.option("dbtable", "orders").option("user", "retail_user")
.option("password", "itversity").load()

similarly i have the same thing to connect my local mysql

spark-shell --jars /usr/share/java/mysql-connector-java.jar
val dataframe_mysql = sqlContext.read.format("jdbc")
.option("url", "jdbc:mysql://mylocalip:3306/employee")
.option("driver", "com.mysql.jdbc.Driver")
.option("dbtable", "emp").option("user", "root")
.option("password", "root").load()

I am getting this exception
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

could any one please help me on this

Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster


#2

Do you have this file in your local system?


#3

Hi @vinodnerella
I am the same user of @raki28
Actually i am connecting my local Db to itversity lab (spark) .
The jar /usr/share/java/mysql-connector-java.jar is contained in itversity lab.

Do I need to have it in local machine (ie Windows machine) like

spark-shell --jars D:\jars\mysql-connector-java.jar

val dataframe_mysql = sqlcontext.read.format("jdbc")
.option("url", "jdbc:mysql://172.25.11.137:3306/employee")
.option("driver", "com.mysql.jdbc.Driver").option("dbtable", "emp")
.option("user", "root").option("password", "root").load()

exception: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

Please advise


#4

@vinodnerella - We cannot connect softwares/applications/IDEs installed on local machines to itversity cluster as we are outside of itversity network. Right?


#5

that’s what I was wondering.


#6

I am not sure on the Windows system. Let me see if I can check this and get back to you.


#7

The following link will help you on how to run application on Itversity Cluster


#8

Hi,

Any update on the connection issue.


#9

Can you give more details? Where you are running spark? And Whether required ports are opened to connect to MySQL instance?
What is the complete process that you are working on?


#10

I am running spark on itversity cluster to access my local DB. Is is possible to connect like that?
If it is possible can you please share how to connect.


#11

Is your local DB accessible over the internet?