Unable to execute flume


#1

This is my example.conf

Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

Describe the sink
a1.sinks.k1.type = hdfs
a2.sinks.k1.hdfs.path =hdfs://gw01.itversity.com/user/ajinkyasaraf13/flume
a1.sinks.k1.filePrefix = netcat

Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channels = c1

and when i run

flume-ng agent --name a1 --conf /home/ajinkyasaraf13/flume/conf/ --conf-file /home/ajinkyasaraf13/flume/conf/example.conf

this is what i get

17/03/02 20:59:12 ERROR lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} } - Exception follows.
org.apache.flume.FlumeException: java.net.BindException: Address already in use
at org.apache.flume.source.NetcatSource.start(NetcatSource.java:168)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.flume.source.NetcatSource.start(NetcatSource.java:162)
… 9 more

am i missing something?


#2

You can try with different port

Also HDFS path is - hdfs://nn01.itversity.com:8020/user/ajinkyasaraf13/flume


#3

thanks a lot…
it worked

but now i dont see any directory called flume
in the given path
/user/ajinkyasaraf13


#4

@Ajinkya_Saraf13
You have to create using mkdir command.
hadoop fs -mkdir /user/ajinkyasaraf13/flume


#5

i have the directory , i meant i dont see the netcat file…sorry about that


#6

i have uploaded screenshots, if it helps


#7

@Ajinkya_Saraf13

You are missing hdfs prior to filePrefix.
hdfs.filePrefix

Refer the manual - https://flume.apache.org/releases/content/1.4.0/FlumeUserGuide.html#hdfs-sink


#8

got it…thanks a lot


#9

Hi,
I followed the same steps, i still get the error Address already in use.

example.conf: A single-node Flume configuration

Name the components on this agent

a1.sources = r1
a1.sinks = k1
a1.channels = c1

Describe/configure the source

a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

Describe the sink

a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path =hdfs://nn01.itversity.com:8020/user/kumsavarthami/flume
a1.sinks.k1.hdfs.filePrefix = netcat

Use a channel which buffers events in memory

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

Bind the source and sink to the channel

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


#10

Change the port number and run again


#11

Name the components on this agent

wh.sources = ws
wh.sinks = k1
wh.channels = mem

Describe/configure the source

wh.sources.ws.type = exec
wh.sources.ws.command = tail -f /opt/gen_logs/logs/access.log

wh.sinks.k1.type = logger

Use a channel which buffers events in memory

wh.channels.mem.type = memory
wh.channels.mem.capacity = 1000
wh.channels.mem.transactionCapacity = 100

Bind the source and sink to the channel

wh.sources.ws.channels = mem
wh.sinks.k1.channel = mem
[kumsavarthami@gw02 flumedemo]$

This is my another example. This is not reading the file

p/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ojdbc6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/paranamer-2.3.ja
r:/usr/hdp/2.6.5.0-292/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-hdfs-plugin-impl:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-hdfs-plugin-shim-0.7.0.2.
6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-plugin-classloader-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-yarn-plugin-impl:/usr/hdp/2.6.5.0-292/hadoop/
lib/ranger-yarn-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292
/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/zookeeper-3.4.6.2.6.5.0-
292.jar:/usr/hdp/2.6.5.0-292/zookeeper/bin:/usr/hdp/2.6.5.0-292/zookeeper/conf:/usr/hdp/2.6.5.0-292/zookeeper/doc:/usr/hdp/2.6.5.0-292/zookeeper/etc:/usr/hdp/2.6.5.0-292/zookeepe
r/lib:/usr/hdp/2.6.5.0-292/zookeeper/man:/usr/hdp/2.6.5.0-292/zookeeper/usr:/usr/hdp/2.6.5.0-292/zookeeper/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/zookeeper/zookeepe
r.jar:/conf:/lib/*’ -Djava.library.path=::/usr/hdp/2.6.5.0-292/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native::/usr/hdp/2.6.5.0-292/hadoop/lib/native/Lin
ux-amd64-64:/usr/hdp/2.6.5.0-292/hadoop/lib/native org.apache.flume.node.Application -n wh -f /home/kumsavarthami/flumedemo/wshdfs.conf
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property ‘log4j2.debug’ to show Log4j2 internal
initialization logging.


#12

Include conf file as part of flume agent
flume-ng agent --name wh --conf-file /path --conf /etc/flume/conf


#13

That worked. Thanks!


#14