Unable to launch sparkui and resource manager


#1

Hi I am unable to launch sparkUI.

Although while starting the spark-shell i have tried different port options

18/06/17 07:54:16 INFO Utils: Successfully started service ‘SparkUI’ on port 12000.
18/06/17 07:54:16 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.16.1.109:12000
18/06/17 07:54:16 INFO Executor: Starting executor ID driver on host localhost
18/06/17 07:54:16 INFO Executor: Using REPL class URI: http://172.16.1.109:55397
18/06/17 07:54:16 INFO Utils: Successfully started service ‘org.apache.spark.network.netty.NettyBlockTransferService’ on port 44044.
18/06/17 07:54:16 INFO NettyBlockTransferService: Server created on 44044
18/06/17 07:54:16 INFO BlockManager: external shuffle service port = 7337
18/06/17 07:54:16 INFO BlockManagerMaster: Trying to register BlockManager
18/06/17 07:54:16 INFO BlockManagerMasterEndpoint: Registering block manager localhost:44044 with 511.1 MB RAM, BlockManagerId(driver, localhost, 44044)
18/06/17 07:54:16 INFO BlockManagerMaster: Registered BlockManager
18/06/17 07:54:17 INFO EventLoggingListener: Logging events to hdfs:///spark-history/local-1529236456137
18/06/17 07:54:17 INFO SparkILoop: Created spark context…
Spark context available as sc.

But while launching from the browser , the SparkUI doesnt connect and gives “Page cannot be displayed error”

Could you please assist here.
Even from the quick links i cant connect to Resource Manager
http://rm01.itversity.com:8088/

Could you please help here

Sign up for our state of the art Big Data cluster for hands on practice as developer. Cluster have Hadoop, Spark, Hive, Sqoop, Kafka and more.



#2

@Sameer_Rao To access resource manager follow the link Accessing resource manager web UI

And change the ip with your gateway name to get access to sparkUI jobs.

Example: gw03.itversity.com:<Assigned port>


#3

Thanks ! this was helpful !
When launching
pyspark --master yarn --conf spark.ui.port=17818 --num-executors 14 --executor-memory 3g --executor-cores 4

i get errors

Caused by: java.nio.channels.ClosedChannelException
18/06/28 00:04:30 ERROR TransportClient: Failed to send RPC 7339423828195556328 to wn03.itversity.com/172.16.1.104:37569: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
18/06/28 00:04:30 WARN NettyRpcEndpointRef: Error sending message [message = RequestExecutors(0,0,Map())] in 1 attempts
java.io.IOException: Failed to send RPC 7339423828195556328 to wn03.itversity.com/172.16.1.104:37569: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801)
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
18/06/28 00:04:33 ERROR TransportClient: Failed to send RPC 6490132699757942167 to wn03.itversity.com/172.16.1.104:37569: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
18/06/28 00:04:33 WARN NettyRpcEndpointRef: Error sending message [message = RequestExecutors(0,0,Map())] in 2 attempts
java.io.IOException: Failed to send RPC 6490132699757942167 to wn03.itversity.com/172.16.1.104:37569: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)