Another instance of Derby may have already booted the database /home/cloudera/metastore_db

I am trying to load data into Hive table using Spark. Below are the first two lines I wrote.

val warehouseLocation = "file:${system:user.dir}/spark-warehouse"
val spark = SparkSession
   .config("spark.sql.warehouse.dir", warehouseLocation)

But the second line is giving me the below exception.

Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@1cf369f7, see the next exception for details.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
    ... 155 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /home/cloudera/metastore_db.

This is the ConnectionURL parameter from my Hive-Site.xml

JDBC connect string for a JDBC metastore

I copied the hive-site.xml to the ‘/usr/lib/spark/conf’ But still the error exists. I am using Spark version: 2.1.1. Could anyone tell me what are the changes I need to do in Hive-site.xml ?

Is it working fine with 1.6?

I don’t think you can use SparkSession in the version 1.x
It is used from 2x

see if another if there are instances of HiveContext is launched
ps -ef | grep spark-shell

also check derby root folders and see if there is any file *.lck and see deleting this file helps

If you have already resolved the issues in another way pls post it here

I can see these processes if I give the command: ps -ef | grep spark-shell

loudera  9622  9607  0 06:45 pts/1    00:00:00 bash /usr/lib/spark/bin/spark-shell
cloudera  9626  9622  1 06:45 pts/1    00:01:04 /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/* -Dscala.usejavacp=true -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name Spark shell spark-shell
cloudera 11838  9402  0 07:57 pts/0    00:00:00 grep spark-shell

There is no ‘Hive’ process here.

ok can you configure custom ui port than taking default and see if shell starts

spark-shell --conf spark.ui.port=5555 (this can be some 4 dig random num)

Also looks like there are two there are multiple sc (spark-shells) in use.ideally only one sc is allowed hence no free ports available try to kill 9607,9622 if there are not in use by any application and try to restart spark-shell