Flume-ng-sdk-1.5.2.2.5.0.0-1245.jar not present on gw03 node


#1

Hi Team

flume-ng-sdk-1.5.2.2.5.0.0-1245.jar is not there under directory /usr/hdp/2.5.0.0-1245/flume/lib
in gw03.itversity.com node.

Due to this my spark flume integration job is generating empty files as an output.

Thanks
Abhishek


#2

@Abhishek_Shukla The location your are referring is old version of HDP. We already upgrade our cluster with HDP 2.6.5.0.

Please refer screenshot to get your required JAR.


#3
  1. filename that was mentioned in git code is different than what you point.
    flume-ng-sdk-1.5.2.2.5.0.0-1245.jar – Git Code
    flume-ng-sdk-1.5.2.2.6.5.0-292.jar – referred in above answer.

  2. considering flume-ng-sdk-1.5.2.2.6.5.0-292.jar filename, I have updated spark-submit as below
    spark-submit
    –class streamingDeptCountFlume
    –master yarn
    –conf spark.ui.port=12986
    –jars “/usr/hdp/2.5.0.0-1245/spark/lib/spark-streaming-flume-sink_2.10-1.6.2.jar,/usr/hdp/2.5.0.0-1245/spark/lib/spark-streaming-flume_2.10-1.6.2.jar,/usr/hdp/2.5.0.0-1245/spark/lib/commons-lang3-3.5.jar,/usr/hdp/current/flume-server/lib/flume-ng-sdk-1.5.2.2.6.5.0-292.jar”
    retail_2.10-0.1.jar yarn-client gw03.itversity.com 8111

Still I am getting file empty generated.


#4

/usr/hdp/current/flume-server/lib/flume-ng-sdk-1.5.2.2.6.5.0-292.jar”

here curernt is a soft link.

use /usr/hdp/2.6.5.0-292/flume/lib/ and it will work.


#5

using below spark-submit to start, still the same problem to have empty file. any idea?

spark-submit
–class streamingDeptCountFlume
–master yarn
–conf spark.ui.port=12986
–jars “/usr/hdp/2.5.0.0-1245/spark/lib/spark-streaming-flume-sink_2.10-1.6.2.jar,/usr/hdp/2.5.0.0-1245/spark/lib/spark-streaming-flume_2.10-1.6.2.jar,/usr/hdp/2.5.0.0-1245/spark/lib/commons-lang3-3.5.jar,/usr/hdp/2.6.5.0-292/flume/lib/flume-ng-sdk-1.5.2.2.6.5.0-292.jar”
retail_2.10-0.1.jar yarn-client gw03.itversity.com 8111