Not able to run the word count program as a application using spark-submit command


File “/home/girishbommisetty/project/csds-spark-emr-master/”, line 24, in
File “/usr/hdp/current/spark-client/python/lib/”, line 1558, in reduceByKey
File “/usr/hdp/current/spark-client/python/lib/”, line 1768, in combineByKey
File “/usr/hdp/current/spark-client/python/lib/”, line 2169, in _defaultReducePartitions
File “/usr/hdp/current/spark-client/python/lib/”, line 2363, in getNumPartitions
File “/usr/hdp/current/spark-client/python/lib/”, line 813, in call
File “/usr/hdp/current/spark-client/python/lib/”, line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o47.partitions.
: java.lang.IllegalArgumentException: user

my code is below:

command used : spark-submit --master yarn | tee output.txt

import sys
from operator import add
from pyspark import SparkConf, SparkContext

#for SparkConf() check out
conf = (SparkConf()
.set(“spark.executor.memory”, “1g”))
sc = SparkContext(conf = conf)

print(“Launch App…”)
if name == “main”:
print(“Initiating main…”)

    inputFile = "hdfs://user/girishbommisetty/input.txt"
    print("Counting words in ", inputFile)
    lines = sc.textFile(inputFile)

    #for lambdas check out
    lines_nonempty = lines.filter( lambda x: len(x) > 0 )
    counts = lines_nonempty.flatMap(lambda x: x.split(' ')) \
                  .map(lambda x: (x, 1)) \
    output = counts.collect()
    for (word, count) in output:
        print("%s: %i" % (word, count))



Give HDFS full path and try now

inputFile = “hdfs://”