SPARK2 not running in gw03.itversity.com

apache-spark

#1

Hi,

Can you please start SPARK2 in gw03

Regards,
Nagaraj


#2

What is the error you are getting?


#3

It says command not found. Also, in ambaari it is showing as down.


#4

Can you share the command you are running?


#5

Following are the commands

pyspark2

spark2-shell


#6

It is not correct.

You have to run

export SPARK_MAJOR_VERSION=2

then spark-shell or pyspark


#7

Thanks. Let me try that. If you have any doc which has these kind of commands, would be very helpful.


#8

Hi,

I am able to connect to SPARK2 after exporting variable. But I get errors when I run some commands.

Following are the commands and error:

spark.catalog.listDatabases()
Traceback (most recent call last):
File “”, line 1, in pyspark
File “/usr/hdp/2.5.0.0-1245/spark2/python/pyspark/sql/catalog.py”, line 61, in listDatabases
iter = self._jcatalog.listDatabases().toLocalIterator()
File “/usr/hdp/2.5.0.0-1245/spark2/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py”, line 933, in call
File “/usr/hdp/2.5.0.0-1245/spark2/python/pyspark/sql/utils.py”, line 69, in deco
raise AnalysisException(s.split(’: ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u’org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:org.apache.hadoop.security.AccessControlException: Permission denied: user=nagaraj_tk, access=EXECUTE, inode="/user/sushmib4/hive/warehouse":sushmib4:hdfs:drwx------\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1827)\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3972)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1130)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:851)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.ap

sc.getConfig().getAll()
Traceback (most recent call last):
File “”, line 1, in


#9

Additional error message:
AttributeError: ‘SparkContext’ object has no attribute ‘getConfig’


#10

getConfig().getAll() is available only in spark-shell not in pyspark.

You can run spark.sql(“show databases”).show to list databases.