Unable to connect Hbase through spark shell

spark-shell
Unable to connect Hbase through spark shell
5.0 1

#1

Hi Team,

Am unable to connect hbase from spark shell. Not able to import Hbase configuration object. Error posted below. Is any additional parameter need to add while launching the shell.
Please let me know and help to solve the issue. If any sample, please post the same.

Command

[anuvenkatesheee@gw03 ~]$ spark-shell --driver-java-options “-Dlog4j.configuration=file:////home/anuvenkatesheee/log4j.properties” --master yarn --conf spark.ui.port=1100 --jars /usr/hdp/2.5.0.0-1245/hbase/lib/hbase-client.jar, /usr/hdp/2.5.0.0-1245/hbase/lib/hbase-common.jar, /usr/hdp/2.5.0.0-1245/hbase/lib/hbase-server.jar, /usr/hdp/2.5.0.0-1245/hbase/lib/hbase-protocol.jar, /usr/hdp/2.5.0.0-1245/hbase/lib/hbase-procedure.jar --files /usr/hdp/2.5.0.0-1245/hbase/conf/hbase-site.xml
SPARK_MAJOR_VERSION is set to 2, using Spark2
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel).
18/04/15 13:34:45 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.
Spark context Web UI available at http://172.16.1.113:1100
Spark context available as ‘sc’ (master = yarn, app id = application_1520592249193_48491).
Spark session available as ‘spark’.
Welcome to
____ __
/ / ___ / /
\ / _ / _ `/ __/ '/
/
/ .__/_,// //_\ version 2.0.0.2.5.0.0-1245
/
/

Using Scala version 2.11.8 (Java HotSpot™ 64-Bit Server VM, Java 1.8.0_77)
Type in expressions to have them evaluated.
Type :help for more information.

Error

scala> import org.apache.hadoop.hbase.HBaseConfiguration
:23: error: object HBaseConfiguration is not a member of package org.apache.hadoop.hbase
import org.apache.hadoop.hbase.HBaseConfiguration


#2

@itversity Dear Durga sir, please have a look and help me to resolve


#3

It will not work that way, we have to use packages or jars so that Spark is aware of respective HBase APIs.

@kmln will help you in this regard.


#4

Hello Venkatesh,

You can use hbase by including the jars hbase-client-1.1.2.jar and hbase-common-1.1.2.jar . You can download the jars from the apache repository.
You can then pass the jars to the spark-shell using the command below:

spark-shell --jars "/home/koushikmln/hbase-client-1.1.2.jar,/home/koushikmln/hbase-common-1.1.2.jar"

Regards,
Koushik


#5

@kmln, @dgadiraju

Dear Koushik,

I tried as per your comments, but still unable to import HBase configuration object.

Spark-shell command

spark-shell --driver-java-options “-Dlog4j.configuration=file:////home/anuvenkatesheee/log4j.properties” --master yarn --conf spark.ui.port=1100 --jars “/home/anuvenkatesheee/hbase-client-1.1.2.jar, /home/anuvenkateshee/hbase-common-1.1.2.jar”

Inside shell

scala> import org.apache.hadoop.hbase.

HbaseConfiguration object not listed in options.

scala> import org.apache.hadoop.hbase.HBaseConfiguration
:23: error: object HBaseConfiguration is not a member of package org.apache.hadoop.hbase
import org.apache.hadoop.hbase.HBaseConfiguration


#6

Hello Venkatesh,

There should not be any space after comma while separating your jar files.

spark-shell --master yarn --conf spark.ui.port=12333 --jars="/usr/hdp/2.5.0.0-1245/hbase/lib/hbase-client.jar,/usr/hdp/2.5.0.0-1245/hbase/lib/hbase-common.jar"

You can see what all jars are being loaded when you start the spark shell on the top. If any jar is not being loaded it will throw an error. You can debug using that.

Regards,
Koushik


#7

@kmln
Thanks Koushik. You are right error resolved after removing spaces while separating jars.
:smile:

Regards
Venkatesh


#8