Flume Spark integration error


#1

Hi Team,

I am getting a connection refused error on gw02.itversity.com error when running flume and spark integration. Below are the commands executed with scripts in the mentioned path:

start flume agent

flume-ng agent -n src -f /home/ankitakalita/StreamingAnaytics/wshdfsstream.conf

submit spark job

spark-submit --master yarn
–conf spark.ui.port=12890
–jars “/usr/hdp/2.5.0.0-1245/spark/lib/spark-streaming-flume_2.10-1.6.2.jar,/usr/hdp/2.5.0.0-1245/spark/lib/spark-streaming-flume-sink_2.10-1.6.2.jar,/usr/hdp/2.5.0.0-1245/flume/lib/flume-ng-sdk-1.5.2.2.5.0.0-1245.jar”
/home/ankitakalita/StreamingAnaytics/streamDeptCountFlume.py gw02.itversity.com 8123 /user/ankitakalita/StreamingDeptCountFlume/cnt

Could you please assist. See snippet of the error message below:

Thanks.


#2

Hi Team,

Could you help on this please…

@vinodnerella @annapurna


#3

In wshdfsstream.conf file the agent name is sdc and you have gaven src.


#4

Hi @annapurna

I changed the agent name but was still getting the error. The data was getting written to the hdfs sink but not to the spark sink.