Unable to launch pyspark in console

#1

Hi @dgadiraju, @itversity1

when I am trying to launch pyspark getting below error

Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
Traceback (most recent call last):
File “/usr/lib64/python2.7/site.py”, line 62, in
import os
File “/usr/lib64/python2.7/os.py”, line 398, in
import UserDict
File “/usr/lib64/python2.7/UserDict.py”, line 83, in
import _abcoll
File “/usr/lib64/python2.7/_abcoll.py”, line 11, in
from abc import ABCMeta, abstractmethod
File “abc.py”, line 1, in
from pyspark import SparkContext,SparkConf
File “/usr/hdp/current/spark-client/python/pyspark/init.py”, line 41, in
from pyspark.context import SparkContext
File “/usr/hdp/current/spark-client/python/pyspark/context.py”, line 21, in
import shutil
File “/usr/lib64/python2.7/shutil.py”, line 12, in
import collections
File “/usr/lib64/python2.7/collections.py”, line 6, in
all += _abcoll.all
AttributeError: ‘module’ object has no attribute ‘all

0 Likes

#2

@rakesdey0018 - Can you please share the command that you are using?

0 Likes

#3

@vinodnerella

‘pyspark’ command only

0 Likes

#4

Where are you running this? Also please share the complete command you are using.

You cannot just use pyspark on our multinode cluster.

0 Likes

#5

I used multiple time in your cluster… faced this issue first time… ‘spark-shell’ command working fine in your cluster

0 Likes

#6

Can you just paste the command you are using so that I can suggest you accordingly?

0 Likes

#7

Below are the thing

[rakeshdey0018@gw02 ~]$ pyspark
Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
Traceback (most recent call last):
File “/usr/lib64/python2.7/site.py”, line 62, in
import os
File “/usr/lib64/python2.7/os.py”, line 398, in
import UserDict
File “/usr/lib64/python2.7/UserDict.py”, line 83, in
import _abcoll
File “/usr/lib64/python2.7/_abcoll.py”, line 11, in
from abc import ABCMeta, abstractmethod
File “abc.py”, line 1, in
from pyspark import SparkContext,SparkConf
File “/usr/hdp/current/spark-client/python/pyspark/init.py”, line 41, in
from pyspark.context import SparkContext
File “/usr/hdp/current/spark-client/python/pyspark/context.py”, line 21, in
import shutil
File “/usr/lib64/python2.7/shutil.py”, line 12, in
import collections
File “/usr/lib64/python2.7/collections.py”, line 6, in
all += _abcoll.all
AttributeError: ‘module’ object has no attribute ‘all

0 Likes

#8

Also tried below command, same error

[rakeshdey0018@gw02 ~]$ pyspark --conf spark.ui.port=12569

0 Likes

#9

You have some python files in your home directory. I have moved those files to archive directory and tried running pyspark. It is working with out any issues.

It is trying to run abc.py when you try to launch pyspark. Not sure why!!!

1 Like

#10

Strange… Interesting :slight_smile:

0 Likes