Cant able to connect sqlContext in Itversity lab

bigdatalabs

#1

Hi,

SqlContext is getting exception while connecting in Itversity lab.
I am connecting this using spark-shell

scala> val sqlcontext = new org.apache.spark.sql.SQLContext(sc)
sqlcontext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@4dd2f2db

scala> val dataframe_mysql = sqlcontext.read.format("jdbc").option("url", "jdbc:mysql://nn01.itversity.com:3306/retail_db").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "orders").option("user", "retail_dba").option("password", "itversity").load()
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
        at scala.tools.nsc.interpreter.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:83)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Could any one please advice me how to connect mysql in spark.


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster


#2

@raki28 Launch spark-shell as below command.

spark-shell --jars /usr/share/java/mysql-connector-java.jar

Try below query

val dataframe_mysql = sqlContext.read.format("jdbc").option("url", "jdbc:mysql://ms.itversity.com:3306/retail_db").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "orders").option("user", "retail_user").option("password", "itversity").load()


#3

Thank you very much @annapurna.


#4