sqlContext not working

#1

Hi,

sqlContext.sql is not working even after “import sqlContext.implicits._”

0 Likes

#2

@itversity ,
But if I run it on HiveContext like below:

import SparkContext._
import org.apache.spark.sql.hive._
val sqlContext = new HiveContext(sc)

I know sqlContext runs with HiveContext or SparkContext(i.e SQLContext) and this new config perfectly makes sense also as HiveContext just extends SparkContext,
but with
“val sqlContext = new SQLContext(sc)”,
it was working sometimes back.
Can anyone please let me know why this sudden discrepancy.

Thanks and Regards,
Sabby

0 Likes

#3

Where are you running this? Is it on our labs or some other environment?
If you are running else where which version of Spark you are running?

Also it will be difficult for us to reproduce the issue and come up with the solution if it is some other environment than our labs.

0 Likes

#4

Hi @itversity ,

I am running it in your lab only.
earlier with SQLContext(i.e SparkContext), it was working.
But now only working with HiveContext.

Thanks and Regards,
Sabby

0 Likes

#5

Can you share the spark-shell command you are using and also complete code you are running?

0 Likes

#6

but if i change the sqlContext from HiveContext to SparkContext, it is not working.

It works with HiveContext, but why not with SparkContext.
Is it because SparkContext is used for core APIs and Spark programing and HiveContext is extending the SparkContext for query purposes?
I want to confirm the reason.

Thanks and Regards,
Sabyasachi

0 Likes

#7

Can you share the spark-shell command as well?

0 Likes

#8

sqlContext of type org.apache.spark.sql.SQLContext will not work with Hive.

This code is working for me and it is always the case.

val sqlContext = new SQLContext(sc)
val orders = sqlContext.read.json("/public/retail_db_json/orders")
orders.registerTempTable("orders")
sqlContext.sql("show tables").show // will not list hive tables
sqlContext.sql("select * from orders").show
0 Likes

#9

@itversity,

I am using the same command:

But still error persists.

Thanks

0 Likes

#10

You need to have tables, then only it will work. You will not be able to get the Information from Hive using SQL Context.

If you want you can try the code shared by me.

0 Likes