Issue with flumes and spark integration


#1

Hello,

While running the spark-submit below is error message that i received.
Warning: Local jar /usr/hdp/2.5.0.0-1245/flume/lib/flume-ng-sdk-1.5.2.2.5.0.0-1245.jar does not exist,
NameError: name ‘FlumesUtils’ is not defined

Please help me with this issue. Thank you


#2

Can you share the spark-submit command that you are using?


#3

spark-submit --master yarn
–conf spark.ui.port=12890
–jars “/usr/hdp/2.5.0.0-1245/spark/lib/spark-streaming-flume_2.10-1.6.2.jar,/usr/hdp/2.5.0.0-1245/spark/lib/spark-streaming-flume-sink_2.10-1.6.2.jar,/usr/hdp/2.5.0.0-1245/flume/lib/flume-ng-sdk-1.5.2.2.5.0.0-1245.jar”
src/main/python/StreamingFlumeDepartmentCount.py
gw02.itversity.com 8123 /user/prabhakarritesh/streamingdept/cnt


#4

@prabhakar.ritesh The path which u have given is incorrect.

Refer the below screenshot.
image