Not an Issue with Labs Error While WorKing With Flume agent

flume

#1


Above I have Attached ScreenShots Regarding Error…
And while starting the flume-ng agent I did it in this way::
cd /usr/hdp/current/flume-server
bin/flume-ng agent -n agent name --conf conf directory --conf-file path for conf file
flume agent had been struck for a long time and I am able to produce messages for kafka topic but I am not able to load data into hdfs

Can Anyone help on my issue and guide me.


#2

@bindu_botlagunta Update your HDFS path hostname. Chage it from gw02.itversity.com to nn01.itversity.com.


#3

Thank you sir,
I have changed the hdfs hostname from gw02.itversity.com to nn01.itversity.com but it was repeating the same.
Even i cant see any errors on console while running flume agent.


#4

@bindu_botlagunta There is some issue with longing in flume. We are working on it, Your flume conf is working fine, I can see the output in the specified location.

Please refer the screenshot.


#5

Thank you sir,

But this is not output of flume,To work on remaining steps of my usecase I used copyFromLocal to get data into the path, Flume is not working…


#6

@bindu_botlagunta

  1. Try below configuration file.

  2. Run the flume agent.
    flume-ng agent --name wk --conf-file /conf-file-path

  3. Then run kafka-console-consumer command in new console.
    kafka-console-consumer.sh --bootstrap-server wn01.itversity.com:6667,wn02.itversity.com:6667,wn03.itversity.com:6667 --topic kafkasourcehdfs < /home/annapurnachinta/data/electionresults/crime_data.csv

  4. output
    hdfs dfs -ls /user/whoami/flume_kafka_demo/