Apache-Flume Error unable to execute - BigData Labs

flume
bigdatalabs

#1

Hi,

I am facing the below error on running Flume Agent(2 Issues), I’m trying to resolve since last 2 days but unable too. Looks like a jar file is missing. Unable to add the Jar to the labs as well via WinSCP.
Could you please take a look into this and resolve at the earliest. This is a blocker for me to proceed further.

I have checked the below source(web server), it is generating the data
tail -F /opt/gen_logs/logs/access.log

Facing issue while running the below commands
Issue: i) On Running: flume-ng agent -n -sdc -f sdc.conf

sdc.conf

# example.conf: A single-node Flume configuration

Name the components on this agent

sdc.sources = ws
sdc.sinks = hd spark
sdc.channels = hdmem sparkmem

Describe/configure the source

sdc.sources.ws.type = exec
sdc.sources.ws.command = tail -F /opt/gen_logs/logs/access.log

Describe the sink

sdc.sinks.hd.type = hdfs
sdc.sinks.hd.hdfs.path = hdfs://nn01.itversity.com:8020/user/jvanchir/flume_demo

sdc.sinks.hd.hdfs.filePrefix = FlumeDemo
sdc.sinks.hd.hdfs.fileSuffix = .txt
sdc.sinks.hd.hdfs.rollInterval = 120
sdc.sinks.hd.hdfs.rollSize = 1048576
sdc.sinks.hd.hdfs.rollCount = 100
sdc.sinks.hd.hdfs.fileType = DataStream

sdc.sinks.spark.type = org.apache.spark.streaming.flume.sink.SparkSink
sdc.sinks.spark.hostname = gw03.itversity.com
sdc.sinks.spark.port = 8123

Use a channel which buffers events in memory

sdc.channels.hdmem.type = memory
sdc.channels.hdmem.capacity = 1000
sdc.channels.hdmem.transactionCapacity = 100

sdc.channels.sparkmem.type = memory
sdc.channels.sparkmem.capacity = 1000
sdc.channels.sparkmem.transactionCapacity = 100

Bind the source and sink to the channel

sdc.sources.ws.channels = hdmem sparkmem
sdc.sinks.hd.channel = hdmem
sdc.sinks.spark.channel = sparkmem

Issue: ii)On running: flume-ng version

Could you also please check the lib folder, looks like there are few more jars missing which are required for the dependency to Integrate Flume, Kafka, and spark streaming.

When I run the below command there are 2 versions of Flume, which one am I supposed to use, because when I compared both each has some Jars missing in both the versions.

[jvanchir@gw03 ~]$ cd /usr/hdp/
[jvanchir@gw03 hdp]$ ls
2.5.0.0-1245 2.6.5.0-292 current

Regards,
Jay


#2

@Jayvardhan_Reddy_Van

Run the flume-ng agent command from the place where conf file exists. I have tested using your console it is working fine.


#3

Hi Sunil,

I am still facing the same issue please check the below screen-shot.
Let me know if I am invoking the flume-ng agent command at the wrong place.
Location where I am executing the command.

Error

Could you also please check the lib folder, looks like there are few more jars missing which are required for the dependency to Integrate Flume, Kafka, and spark streaming that I came accross as part of CCA-175 videos as well.

When I run the below command there are 2 versions of Flume, which one am I supposed to use, because when I compared both each has some Jars missing in both the versions.

[jvanchir@gw03 ~]$ cd /usr/hdp/
[jvanchir@gw03 hdp]$ ls
2.5.0.0-1245 2.6.5.0-292 current


#4

Ignore all other exceptions just check whether the file is created or not in the given hdfs path /user/jvanchir/flume_demo

image

You can see the file is created in the given path.


#5

Thanks Sunil,
It is working as expected, now the logs also seem to run as expected, earlier on executing the flume command the execution would get hung after the exception in loggers as shown in the previous screen-shots and the execution wouldn’t proceed beyond that point.


#6