Pyspark application runs slow on cluster but runs like charm on local node, Please help

I tried debugging all the possible solutions but unable to run this and scale this on cluster as i need to process 100 million records, This script runs very well on local node as expected but fails to run on cloudera amazon cluster. Please help. Here is the sample data that works on local node - download this sheet as csv

And am i also using this stanford api ( )for this use case. Download link follows, Unzip it and refer the path in the script.

Please help

#spark-submit --master local[*] --packages com.databricks:spark-csv_2.10:1.5.0

from pyspark.sql.types import StringType
from pyspark.sql.functions import udf
from nltk.tag import StanfordNERTagger
import os
from pyspark import SparkFiles
from pyspark import SparkContext, SparkConf
from pyspark.sql import Row
from pyspark.context import SparkContext
from pyspark.sql import HiveContext
from pyspark.sql.functions import udf
from pyspark.sql import SQLContext
def stanford(str):

from stanfordcorenlp import StanfordCoreNLP
nlp = StanfordCoreNLP(r"/home/PSL/sunil.dhage/Downloads/stanford-corenlp-full-2017-06-09")
organizations = []
#organization = ""
output = nlp.ner(str)
organizations = []
organization = ""
for t in output:
    #The word

    word = t[0]
    #What is the current tag
    tag = t[1]
    print(word, tag)
    #If the current tag is the same as the previous tag Append the current word to the previous word
    if (tag == "ORGANIZATION"):
            organization += " " + word
final = "-".join(organizations)
return final

stanford_lassification = udf(stanford, StringType())
#Set context
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)
#Get data
df =‘com.databricks.spark.csv’).options(header=‘true’, inferschema=‘true’).load(r"/home/PSL/sunil.dhage/Downloads/authors_data.csv")

#Create new dataframe with new column organization
df = df.withColumn(“organizations”, stanford_lassification(df[‘affiliation_string’]))

#Save result

Try to process some other data and see if the issue persists!!!

It works very well if i just use the udf with simple logic like appending text to the string of the udf without using external jars and files. The problem seems to be only when i use the external files. It takes 26 mins to process 1000 rows thats very slow on cluster. The code works and gives the output but problem is slowness on the cluster

Do you mean when you use packages, it is slow?