Spark-shell sqlcontext error


#1

WHiles starting Spark shell from console

Getting following exception
8/11/29 22:10:44 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:12654: java.net.BindExceptio
n: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265
)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:252)
at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2040

also sqlcontext giving error
cala> sc
res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@31940d6b
scala> sqlcontext
:26: error: not found: value sqlcontext
sqlcontext
^
Please advice
Thanks,


#2

@S11 Launch spark as below command:

spark-shell --master yarn --conf spark.ui.port=12888 spark.port.maxRetries=100 --num-executors 1 --executor-memory 512M

scala> sqlcontext
:26: error: not found: value sqlcontext
sqlcontext
^

There is a typo in your command. It’s sqlContext .