Conf file error - Spark Streaming with Flume :


#1

Hello Itversity-

Video : 22 Flume and Spark Streaming - Department Wise Count - Setup Flume Agent

I have created the below file:

Name the components on this agent

sdc.sources = ws
sdc.sinks = hd spark
sdc.channels = hdmem sparkmem

Describe/configure the source

sdc.sources.ws.type = exec
sdc.sources.ws.command = tail -F /opt/gen_logs/logs/access.log

Describe the sink

sdc.sinks.hd.type = hdfs
sdc.sinks.hd.hdfs.path = hdfs://nn01.itversity.com:8020/user/dasjeevan/flume_demo

sdc.sinks.hd.filePrefix = FlumeDemo
sdc.sinks.hd.fileSuffix = .txt
sdc.sinks.hd.hdfs.fileType = DataStream

sdc.sinks.hd.hdfs.rollInterval = 120
sdc.sinks.hd.hdfs.rollSize = 1048576
sdc.sinks.hd.hdfs.rollCount = 100

#copied from the below link
#https://spark.apache.org/docs/1.6.3/streaming-flume-integration.html ---- Configuration file

sdc.sinks.spark.type = org.apache.spark.streaming.flume.sink.SparkSink
sdc.sinks.spark.hostname = gw03.itversity.com
sdc.sinks.spark.port = 8123

Use a channel sdcich buffers events in memory

sdc.channels.hdmem.type = memory
sdc.channels.hdmem.capacity = 1000
sdc.channels.hdmem.transactionCapacity = 100

sdc.channels.sparkmem.type = memory
sdc.channels.sparkmem.capacity = 1000
sdc.channels.sparkmem.transactionCapacity = 100

Bind the source and sink to the channel

sdc.sources.ws.channels = hdmem sparkmem
sdc.sinks.hd.channel = hdmem
sdc.sinks.spark.channel = sparkmem

flume command to submit : [dasjeevan@gw03 strdeptcount]$ flume-ng agent -n sdc -f sdc.conf

Error I am receiving

2018-08-16 08:21:02,954 main ERROR Unable to create file /var/log/flume-ng/flume.log java.io.IOException: Could not create directory /var/log/flume-ng
at org.apache.logging.log4j.core.util.FileUtils.mkdir(FileUtils.java:127)
at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:144)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:627)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)

2018-08-16 08:21:02,969 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:229)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at org.apache.logging.log4j.core.co

2018-08-16 08:21:03,304 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)] Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load sink type: org.apache.spark.streaming.flume.sink.SparkSink, class: org.apache.spark.streaming.flume.sink.SparkSink
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69) ~[flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41) ~[flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415) ~[flume-ng-node-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103) ~[flume-ng-node-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) [flume-ng-node-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.flume.sink.SparkSink
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_151]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_151]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_151]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_151]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]
at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67) ~[flume-ng-core-1.5.2.2.6.5.0-292.jar:1.5.2.2.6.5.0-292]
… 11 more

^C2018-08-16 08:22:27,582 (agent-shutdown-hook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:79)] Stopping lifecycle supervisor 20
2018-08-16 08:22:27,583 (agent-shutdown-hook) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop(PollingPropertiesFileConfigurationProvider.java:83)] Configuration provider stopping

========

Can you please help me on it at earliest ?

Regards,
Jeevan


#2

@Jeevan_Das We have added a jar file which is required for your configuration. Try now to run the conf.


#3

Hi,

Even I am facing the same error:
“2018-09-12 20:30:03,696 main ERROR Unable to create file /var/log/flume-ng/flume.log java.io.IOException: Could not create directory /var/log/flume-ng”

Could you please tell which jar file you have added?
Please let me know if you need other information.

Thanks,
Moumita