ERROR hdfs.HDFSEventSink: process failed

17/04/26 03:35:58 ERROR hdfs.HDFSEventSink: process failed
java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:228)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:432)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:380)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
17/04/26 03:35:58 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:228)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:432)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:380)
… 3 more

FIle:

Name the components on this agent

a1.sources = r1
a1.sinks = k1
a1.channels = c1

Describe/configure the source

a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /opt/gen_logs/logs/access.log
a1.sources.r1.channels = c1

Describe the sink

a1.sinks.k1.type = hdfs

Use a channel which buffers events in memory

a1.channels.c1.type = FILE
a1.channels.c1.capacity = 20000
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.checkpointInterval 300000

Customizing Sink for HDFS

a1.sinks.k1.hdfs.path = hdfs://quickstart.cloudera:8020/user/cloudera/flume/%y-%m-%d
a1.sinks.k1.hdfs.filePrefix = flume-%y-%m-%d
a1.sinks.k1.hdfs.rollSize = 1048576
a1.sinks.k1.hdfs.rollCount = 100
a1.sinks.k1.hdfs.rollInterval = 120
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.idleTimeout = 10
ai.sinks.k1.hdfs.useLocalTimeStamp = true

Bind the source and sink to the channel

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

Command:
flume-ng agent --name a1 --conf /home/cloudera/flume/conf/ --conf-file /home/cloudera/flume/conf/example.conf

Getting the same error again and again. Can someone please help pointing out the mistake.
Thanks.

Have you started the logs by using start_logs. Check it with
hadoop fs -tail -F /opt/gen_logs/logs/access.log
If it should returns some logs continuously.

Thanks @N_Chakote for the response but I got the solution. It was due to a small typo in the conf file.

1 Like

Hey can you please tell me how you have fixed this error.