Flume Spark integration error


Hi Team,

I am getting a connection refused error on gw02.itversity.com error when running flume and spark integration. Below are the commands executed with scripts in the mentioned path:

start flume agent

flume-ng agent -n src -f /home/ankitakalita/StreamingAnaytics/wshdfsstream.conf

submit spark job

spark-submit --master yarn
–conf spark.ui.port=12890
–jars “/usr/hdp/,/usr/hdp/,/usr/hdp/”
/home/ankitakalita/StreamingAnaytics/streamDeptCountFlume.py gw02.itversity.com 8123 /user/ankitakalita/StreamingDeptCountFlume/cnt

Could you please assist. See snippet of the error message below:



Hi Team,

Could you help on this please…

@vinodnerella @annapurna


In wshdfsstream.conf file the agent name is sdc and you have gaven src.


Hi @annapurna

I changed the agent name but was still getting the error. The data was getting written to the hdfs sink but not to the spark sink.