pyspark.sql.utils.AnalysisException: u"Table or view not found"

#1

I am trying to access Hive table from a Spark Session but am getting the following exception,
yspark.sql.utils.AnalysisException: u"Table or view not found: srikarec_retail_db.orders;;\n’UnresolvedRelation srikarec_retail_db.orders\n"
19/06/10 11:59:07 INFO SparkContext: Invoking stop() from shutdown hook
19/06/10 11:59:07 INFO SparkUI: Stopped Spark web UI at http://gw02.itversity.com:12506
19/06/10 11:59:07 INFO YarnClientSchedulerBackend: Interrupting monitor thread
19/06/10 11:59:07 INFO YarnClientSchedulerBackend: Shutting down all executors
19/06/10 11:59:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
19/06/10 11:59:07 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,

0 Likes

#2

@srikarEC Could you please reply with the code which you are trying so that i can help you in a better way.

0 Likes

#3

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.window import *
from pyspark.sql import HiveContext
import sys
spark = SparkSession.
builder.
appName(‘Get top 5 customers’).
master(‘yarn-client’).
getOrCreate()
orderpath = sys.argv[1]
orderItemsPath = sys.argv[2]
customersPath = sys.argv[3]
spec=Window.
partitionBy(substring(‘order_date’,1,7), ‘cust_id’)
spec1= Window.
partitionBy(‘order_month’).
orderBy(‘order_month’, desc(‘order_value’))
orders = spark.read.table(‘srikarec_retail_db.orders’)
orderItems = spark.read.table(‘srikarec_retail_db.order_items’)
customers = spark.read.table(‘srikarec_retail_db.customers’)
joinOrdItmCust = orders.
join(orderItems, orders.order_id == orderItems.order_item_id).
join(customers, orders.order_cust_id == customers.cust_id).
select(‘cust_id’,substring(‘orders.order_date’,1,7).alias(‘order_month’),
‘cust_fname’, ‘cust_lname’,
sum(‘order_item_subtotal’).over(spec).alias(‘order_value’)).
distinct().
withColumn(‘rank’, rank().over(spec1)).
withColumn(‘row_number’, row_number().over(spec1)).
filter(“row_number <= 5”)

joinOrdItmCust.
select(‘order_month’,‘cust_id’,‘cust_fname’,‘cust_lname’,‘order_value’).
createOrReplaceTempView(‘exercise1’).
sqlContext.sql(‘create table srikarec_retail_db.exercise_result1 as
select * from exercise1’)

0 Likes