Unable to get the tracking URL spark shell launched

Hi
When i launch the spark with the following command , i dont get the way presented in the course. such as tracking url etc., Can you please check and let me know how do i get that. Thanks

[yashwanth232@gw03 data]$ spark-shell --master yarn \

–deploy-mode client
–conf spark.ui.port=12335
–num-executors 1
–executor-memory 2048M
Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
Welcome to
____ __
/ / ___ / /
\ / _ / _ `/ __/ '/
/
/ .__/_,// //_\ version 1.6.3
/
/
Using Scala version 2.10.5 (Java HotSpot™ 64-Bit Server VM, Java 1.8.0_77)
Type in expressions to have them evaluated.
Type :help for more information.
spark.yarn.driver.memoryOverhead is set but does not apply in client mode.
Spark context available as sc.
Sat Nov 09 12:06:46 EST 2019 Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)


Sat Nov 09 12:06:47 EST 2019:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.12.1.1 - (1704137): instance a816c00e-016e-5121-e6e1-00002fc9eba0
on database directory /tmp/spark-65c7682a-3a07-42e7-bdc8-2586a4cda2ed/metastore with class loader sun.misc.Launcher$AppClassLoader@5c647e05
Loaded from file:/usr/hdp/2.6.5.0-292/spark/lib/spark-assembly-1.6.3.2.6.5.0-292-hadoop2.7.3.2.6.5.0-292.jar
java.vendor=Oracle Corporation
java.runtime.version=1.8.0_77-b03
user.dir=/data
os.name=Linux
os.arch=amd64
os.version=3.10.0-957.27.2.el7.x86_64
derby.system.home=null
Database Class Loader started - derby.database.classpath=’’
SQL context available as sqlContext.

Goodnight.

It gives you said permission denied as a derby client because you are running spark-shell in a directory that you have access to but will not find the address or path where it is derby. Therefore, correct it in the base directory, / home / user_name.

I hope it has been helpful.
Regards.
1

Hi,
I am still not able to get. Can you please help me what’s the path I would need to set in my base directory. Log as below :

Thanks

[yashwanth232@gw03 ~]$ cd /data
-conf spark.ui.port=12335 --num-executors 1 --executor-memory 2048Mnt –
Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
Welcome to
____ __
/ / ___ / /
\ / _ / _ `/ __/ '/
/
/ .__/_,// //_\ version 1.6.3
/
/
Using Scala version 2.10.5 (Java HotSpot™ 64-Bit Server VM, Java 1.8.0_77)
Type in expressions to have them evaluated.
Type :help for more information.
spark.yarn.driver.memoryOverhead is set but does not apply in client mode.
Spark context available as sc.
Sat Nov 09 20:49:46 EST 2019 Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)

Sat Nov 09 20:49:47 EST 2019:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.12.1.1 - (1704137): instance a816c00e-016e-5300-b7cc-00002f7b0f20
on database directory /tmp/spark-699e47c1-68e5-4f8f-b261-45ce47738c3a/metastore with class loader sun.misc.Launcher$AppClassLoader@5c647e05
Loaded from file:/usr/hdp/2.6.5.0-292/spark/lib/spark-assembly-1.6.3.2.6.5.0-292-hadoop2.7.3.2.6.5.0-292.jar
java.vendor=Oracle Corporation
java.runtime.version=1.8.0_77-b03
user.dir=/data
os.name=Linux
os.arch=amd64
os.version=3.10.0-957.27.2.el7.x86_64
derby.system.home=null
Database Class Loader started - derby.database.classpath=’’
SQL context available as sqlContext.
scala>

First put:
cd and press enter.

Then command them:
spark-shell --master yarn –deploy-mode client –conf spark.ui.port = 12335 –num-executors 1 –executor-memory 2048M

Then press enter.

If you want to see the address where you are, enter pwd and press enter.

First put:
cd and press enter.

Then command them:
spark-shell --master yarn –deploy-mode client –conf spark.ui.port = 12335 –num-executors 1 –executor-memory 2048M

Then press enter.

If you want to see the address where you are, enter pwd and press enter.

Then raise the data with the command
sc.textFile (‘path’)

I hope it is useful for you.

Regards.

@Yashwanth

When you execute the spark command it will generate automatically a url which is called as tracking url.

Our admin has disabled extensive logging due to which the tracking url is not generated. You can go to resource manager and track from there.
Here is the url.
http://rm01.itversity.com:19088/cluster