HIVE(https://spark.apache.org/docs/2.4.0/sql-data-sources-hive-tables.html)

when I was reading the Spark-SQL documentation for spark 2.0. on the page it is mentioned
’ Note that the hive.metastore.warehouse.dir property in hive-site.xml is deprecated since Spark 2.0.0. Instead, use spark.sql.warehouse.dir to specify the default location of database in warehouse.’

Can any one explain this, so when we create a hive table using spark.sql(), the metastore is stored in apps/hive/warehouse right?

Practice hive on state of the art Big Data cluster - https://labs.itversity.com
You can sign up for our courses on Udemy using $10 coupons - Udemy Coupons - Big Data Courses


You need to configure the warehouse directory as well.

spark2-shell \
--master yarn \
--conf spark.ui.port=0 \
--conf spark.sql.warehouse.dir=/user/${USER}/warehouse

Please go through the below links


Thanks so much for the help. i found an article on medium by Durga which also help me.