Jar file missing for Flume and Spark Streaming Integration


Hi Team,

I am trying to get flume data to Spark sink and use it to get department count for gen_logs.
As mentioned in the tutorial, we need to mention jars in spark submit command.
I was using below command:

spark-submit --master yarn  \
  --conf spark.ui.port=12890 \
  --jars "/usr/hdp/,/usr/hdp/,/usr/hdp/"  \
  /home/anujidgupta/python_demo/streamingFlumeDeptCount.py \
  gw02.itversity.com 8123 \

Please note flume-ng-sdk- jar is present in different version (/usr/hdp/ as other required jars are present under /usr/hdp/

WHile running I am getting empty files created.
Please suggest if issue is due to jar file location or some other issue.



Not sure. I need to troubleshoot further on this. It could be due to Spark 1.6.3 or some missing jars.