Error: sqlContext not found

apache-spark

#1

Hi,

I get below error when i start spark-shell.

Command:

spark-shell --master yarn --conf spark.ui.port=41234 --packages com.databricks:spark-avro_2.10:2.0.1 --num-executors 2

18/07/12 13:21:14 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@7b52b18a: java.net.BindException: Address already in use
java.net.BindException: Address already in use

18/07/12 13:21:18 INFO Client:
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1531416077613
final status: UNDEFINED
tracking URL: http://rm01.itversity.com:19288/proxy/application_1528589352821_22040/
user: satishp38
18/07/12 13:21:19 INFO Client: Application report for application_1528589352821_22040 (state: ACCEPTED)
18/07/12 13:21:19 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
18/07/12 13:21:19 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> rm01.itversity.com, PROXY_URI_BASES -> http://rm01.itversity.com:19288/proxy/application_1528589352821_22040), /proxy/application_1528589352821_22040
18/07/12 13:21:19 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
18/07/12 13:21:20 INFO Client: Application report for application_1528589352821_22040 (state: RUNNING)
18/07/12 13:21:20 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.16.1.104
ApplicationMaster RPC port: 0
queue: default
start time: 1531416077613
final status: UNDEFINED
tracking URL: http://rm01.itversity.com:19288/proxy/application_1528589352821_22040/
user: satishp38
18/07/12 13:21:20 INFO YarnClientSchedulerBackend: Application application_1528589352821_22040 has started running.
18/07/12 13:21:20 INFO Utils: Successfully started service ‘org.apache.spark.network.netty.NettyBlockTransferService’ on port 36026.
18/07/12 13:21:20 INFO NettyBlockTransferService: Server created on 36026
18/07/12 13:21:20 INFO BlockManager: external shuffle service port = 7337
18/07/12 13:21:20 INFO BlockManagerMaster: Trying to register BlockManager
18/07/12 13:21:20 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.1.113:36026 with 511.1 MB RAM, BlockManagerId(driver, 172.16.1.113, 36026)
18/07/12 13:21:20 INFO BlockManagerMaster: Registered BlockManager
18/07/12 13:21:20 INFO EventLoggingListener: Logging events to hdfs:///spark-history/application_1528589352821_22040
java.lang.OutOfMemoryError: unable to create new native thread

java.lang.NullPointerException
:16: error: not found: value sqlContext
import sqlContext.implicits._
^
:16: error: not found: value sqlContext
import sqlContext.sqlstrong text


#2

Hi @Satish_Paraskar,

It is working fine. Check the below screenshot for your reference.

save

Thanks and Regards,
ITVersity Support Team.