Cannot use SparkContext in pyspark


#1

Can anyone please tell me what is missing here? I cannot use SparkContext in pyspark, here is the detail:

from pyspark import SparkContext, SparkConf
sc =SparkContext()
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server
Traceback (most recent call last):
File “/usr/hdp/2.5.0.0-1245/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py”, line 690, in start
self.socket.connect((self.address, self.port))
File “/usr/lib64/python2.7/socket.py”, line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
Traceback (most recent call last):
File “”, line 1, in
File “/usr/hdp/2.5.0.0-1245/spark/python/pyspark/context.py”, line 115, in init
conf, jsc, profiler_cls)
File “/usr/hdp/2.5.0.0-1245/spark/python/pyspark/context.py”, line 124, in _do_init
self._conf = conf or SparkConf(_jvm=self._jvm)
File “/usr/hdp/2.5.0.0-1245/spark/python/pyspark/conf.py”, line 106, in init
self._jconf = _jvm.SparkConf(loadDefaults)
File “/usr/hdp/2.5.0.0-1245/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py”, line 1188, in getattr
File “/usr/hdp/2.5.0.0-1245/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py”, line 624, in send_command
File “/usr/hdp/2.5.0.0-1245/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py”, line 579, in _get_connection
File “/usr/hdp/2.5.0.0-1245/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py”, line 585, in _create_connection
File “/usr/hdp/2.5.0.0-1245/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py”, line 697, in start
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server


#2

Here is the output:

[paslechoix@gw01 ~]$ pyspark
Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set

18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@1b326567: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4041: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@534189db: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4042: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@32701c24: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4043: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@4f90958b: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4044: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@1c6a95d3: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4045: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@658cf02b: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:44 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4046: java.net.BindException: Address already in use
java.net.BindException: Address already in use
18/02/09 06:54:45 ERROR SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: Service ‘SparkUI’ failed after 16 retries! Consider explicitly setting the appropriate port for the service ‘SparkUI’ (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server


Can anyone help to fix the "error"? I don't think there is an error but it does
#3

I am also getting the same error, can anyone help


#4

@meetarpit & @paslechoix

Please try in Jupyter notebooks & post here if still issue persists.
If notebooks can pickup properly with help of Kernels then issues is with small env settings on nodes.