Getting below error when running flume config file

flume

#1

[sonaliswami8@gw03 ~]$ flume-ng agent --name agent1 --conf-file /home/sonaliswami8/example.conf

flume-config - file

agent1.sinks = hdfs-sink1_1
agent1.sources = source1_1
agent1.channels = fileChannel1_1

agent1.channels.fileChannel1_1.type = file
agent1.channels.fileChannel1_1.capacity = 200000
agent1.channels.fileChannel1_1.transactionCapacity = 1000

agent1.sources.source1_1.type = spooldir

agent1.sources.source1_1.spoolDir = /user/sonaliswami8/desktop/flume_test
agent1.sources.source1_1.fileHeader = true
agent1.sources.source1_1.fileHeaderKey = file
agent1.sinks.hdfs-sink1_1.type = hdfs

agent1.sinks.hdfs-sink1_1.hdfs.path = /user/sonaliswami8/desktop/hdfs/flume
agent1.sinks.hdfs-sink1_1.hdfs.batchSize = 1000
agent1.sinks.hdfs-sink1_1.hdfs.rollSize = 2000
agent1.sinks.hdfs-sink1_1.hdfs.rollInterval = 0
agent1.sinks.hdfs-sink1_1.hdfs.rollCount = 500
agent1.sinks.hdfs-sink1_1.hdfs.writeFormat=Text

agent1.sinks.hdfs-sink1_1.hdfs.fileType = DataStream
agent1.sources.source1_1.channels = fileChannel1_1
agent1.sinks.hdfs-sink1_1.channel = fileChannel1_1

=================================================================================================================
ERROR

13:00:42.784 [lifecycleSupervisor-1-2] ERROR org.apache.flume.lifecycle.LifecycleSupervisor - Unable to start EventDrivenSourceRunner: { source:Spool Direct
ory source source1_1: { spoolDir: /user/sonaliswami8/desktop/flume_test } } - Exception follows.
java.lang.IllegalStateException: Directory does not exist: /user/sonaliswami8/desktop/flume_test
at com.google.common.base.Preconditions.checkState(Preconditions.java:145) ~[guava-11.0.2.jar:?]
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.(ReliableSpoolingFileEventReader.java:139) ~[flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.(ReliableSpoolingFileEventReader.java:75) ~[flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader$Builder.build(ReliableSpoolingFileEventReader.java:648) ~[flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.source.SpoolDirectorySource.start(SpoolDirectorySource.java:99) ~[flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44) ~[flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) [flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5
.0-292]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

What should be the hdfs path and it is even unable to locate the folder but it is present in hdfs?


#2

@sonali1

Can u share the link from where did you follow this config file?