Unable to run flume job for HDFS Sink for Webserver Logs

flume
#1

Hi Team,

I am using the example mentioned in CCA 175 course for loading web server data to HDFS sink using flume.
While my FLume commands seems to be executed with some errors and warning, but in the defined target path, no data has been loaded.
Below is the conf file I used:

=======================

example.conf: A single-node Flume configuration

# Name the components on this agent

wh.sources = ws
wh.sinks = hd
wh.channels = mem

Describe/configure the source

wh.sources.ws.type = exec
wh.sources.ws.command = tail -F /opt/gen_logs/logs/access.log

# Describe the sink

wh.sinks.hd.type = hdfs
wh.sink.hd.hdfs.path = hdfs://nn01.itversity.com:8020/user/anujidgupta/flumedemo

# Use a channel which buffers events in memory

wh.channels.mem.type = memory
wh.channels.mem.capacity = 1000
wh.channels.mem.transactionCapacity = 100

# Bind the source and sink to the channel

wh.sources.ws.channels = mem
wh.sinks.hd.channel = mem

Below is flume command that I used:
[anujidgupta@gw02 wslogstohdfs]$ flume-ng agent --name wh \

–conf-file /home/anujidgupta/flume_demo/wslogstohdfs/wshdfs_hsink.conf
–conf /etc/flume/conf

Also Please find below some snippet of screen output after running Flume command:

2019-05-11 03:46:24,773 main ERROR RollingFileManager (/tmp/flume/flume.log) java.io.FileNotFoundException: /tmp/flume/flume.log (Permission denied) java.io.FileN
otFoundException: /tmp/flume/flume.log (Permission denied)

2019-05-11 03:46:24,778 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.
lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@5552768b] unable to create
manager for [/tmp/flume/flume.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@3c947bc5[pattern=/tmp/flume/archive/f
lume.log.%d{yyyyMMdd}-%i, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=104857600), Cro
nTriggeringPolicy(schedule=0 0 0 * * ?)]), strategy=DefaultRolloverStrategy(min=1, max=20, useMax=true), advertiseURI=null, layout=%d{dd MMM yyyy HH:mm:ss,SSS} %-
5p [%t] (%C.%M:%L) %equals{%x}{[]}{} - %m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core
.appender.rolling.RollingFileManager$RollingFileManagerFactory@5552768b] unable to create manager for [/tmp/flume/flume.log] with data [org.apache.logging.log4j.c
ore.appender.rolling.RollingFileManager$FactoryData@3c947bc5[pattern=/tmp/flume/archive/flume.log.%d{yyyyMMdd}-%i, append=true, bufferedIO=true, bufferSize=8192,
policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=104857600), CronTriggeringPolicy(schedule=0 0 0 * * ?)]), strategy=DefaultRolloverStrate
gy(min=1, max=20, useMax=true), advertiseURI=null, layout=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %equals{%x}{[]}{} - %m%n, filePermissions=null, fileOw
ner=null]]

2019-05-11 03:46:24,780 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: ja
va.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No fa
ctory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender

2019-05-11 03:46:24,782 main ERROR Null object returned for RollingFile in Appenders.
2019-05-11 03:46:24,912 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfiguratio
nProvider.java:61)] Configuration provider starting
2019-05-11 03:46:24,916 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFil
eConfigurationProvider.java:133)] Reloading configuration file:/home/anujidgupta/flume_demo/wslogstohdfs/wshdfs_hsink.conf
2019-05-11 03:46:24,923 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1019)] Proces
sing:hd

2019-05-11 03:46:24,974 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)]
Channel mem connected to [ws]
2019-05-11 03:46:24,979 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sour
ceRunners:{ws=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:ws,state:IDLE} }} sinkRunners:{} channels:{mem=org.apache.flume.channel.Me
moryChannel{name: mem}} }
2019-05-11 03:46:24,985 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel mem
2019-05-11 03:46:24,985 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitor
ed counter group for type: CHANNEL, name: mem: Successfully registered new MBean.
2019-05-11 03:46:24,986 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component t
ype: CHANNEL, name: mem started
2019-05-11 03:46:24,986 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source ws
2019-05-11 03:46:24,986 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:163)] Exec source starting with command: tail -
F /opt/gen_logs/logs/access.log
2019-05-11 03:46:24,987 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitor
ed counter group for type: SOURCE, name: ws: Successfully registered new MBean.
2019-05-11 03:46:24,987 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component t
ype: SOURCE, name: ws started

0 Likes

#2

Hi Team,

Can you please help me on issue mentioned above.

0 Likes

#3

Issue resolved. Mistake in conf file

0 Likes

closed #4
0 Likes