Not able to start flume agent

apache-spark

#1

I am getting the below error when I am using the command flume-ng agent --name a1 --conf-file /home/gangadharanrahul/flume_demo/example.conf

My username is gangadharanrahul

It is initially showing the Warning: JAVA_HOME is not set!

Below is the messages at the end.

.0-292.jar:/usr/hdp/2.6.5.0-292/zookeeper/zookeeper.jar:/conf:/lib/*’ -Djava.library.path=::/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop
/lib/native::/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native org.apache.flume.node.Application --name a1 --conf-file /home/ganga
dharanrahul/flume_demo/example.conf
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property ‘log4j2.debug’ to show Log4j2 int
ernal initialization logging.


#2

@Rahul_Gangadharan Refer this link Flume ERROR StatusLogger No log4j2 configuration file found


#3

Thanks, that worked. But now i am getting the below error

/usr/hdp/2.6.5.0-292/flume/bin/flume-ng.distro: line 247: /usr/lib/jvm/java-1.6.0/bin/java: No such file or directory

Below are the directories in the jvm directory -

drwxr-xr-x 3 root root 4096 Sep 20 2017 java-1.7.0-openjdk-1.7.0.141-2.6.10.1.el7_3.x86_64
lrwxrwxrwx 1 root root 51 Jan 12 2018 jre-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64 -> java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64/jre
drwxr-xr-x 4 root root 4096 Jan 12 2018 java-1.7.0-openjdk-1.7.0.161-2.6.12.0.el7_4.x86_64
lrwxrwxrwx 1 root root 54 Jan 12 2018 jre-1.7.0-openjdk-1.7.0.161-2.6.12.0.el7_4.x86_64 -> java-1.7.0-openjdk-1.7.0.161-2.6.12.0.el7_4.x86_64/jre
lrwxrwxrwx 1 root root 27 Jan 12 2018 jre-1.7.0 -> /etc/alternatives/jre_1.7.0
lrwxrwxrwx 1 root root 35 Jan 12 2018 jre-1.7.0-openjdk -> /etc/alternatives/jre_1.7.0_openjdk
lrwxrwxrwx 1 root root 29 Jan 12 2018 jre-openjdk -> /etc/alternatives/jre_openjdk
lrwxrwxrwx 1 root root 21 Jan 12 2018 jre -> /etc/alternatives/jre
lrwxrwxrwx 1 root root 27 Jan 12 2018 jre-1.8.0 -> /etc/alternatives/jre_1.8.0
lrwxrwxrwx 1 root root 35 Jan 12 2018 jre-1.8.0-openjdk -> /etc/alternatives/jre_1.8.0_openjdk
drwxr-xr-x 7 root root 4096 Jan 12 2018 java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64
lrwxrwxrwx 1 root root 26 Jan 12 2018 java -> /etc/alternatives/java_sdk
lrwxrwxrwx 1 root root 34 Jan 12 2018 java-openjdk -> /etc/alternatives/java_sdk_openjdk
lrwxrwxrwx 1 root root 40 Jan 12 2018 java-1.8.0-openjdk -> /etc/alternatives/java_sdk_1.8.0_openjdk
lrwxrwxrwx 1 root root 32 Jan 12 2018 java-1.8.0 -> /etc/alternatives/java_sdk_1.8.0

How i need to update the flume config file to point to correct path?


#4

Can you share the example.conf file and the flume agent command that you have tried?


#5

example.conf file -

example.conf: A single-node Flume configuration

Name the components on this agent

a1.sources = r1
a1.sinks = k1
a1.channels = c1

Describe/configure the source

a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

Describe the sink

a1.sinks.k1.type = logger

Use a channel which buffers events in memory

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

Bind the source and sink to the channel

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

flume command -

flume-ng agent --name a1 --conf-file /home/gangadharanrahul/flume_demo/example.conf --conf /etc/flume/conf


#6

You can ignore the errors and check whether the telnet is connected or not by sending some random messages.


#7

Yes, it is working. Telnet is connected eventhough it was showing errors while launching. Thanks!