Flume and Spark streaming- Error in Setting up flume agent


#1

Hi team,

Below is my config file for flume using spark and hdfs as sink:-

example.conf: A single-node Flume configuration

Name the components on this agent

sdc.sources = ws
sdc.sinks = hd spark
sdc.channels = hdmem sparkmem

Describe/configure the source

sdc.sources.ws.type = exec
sdc.sources.ws.command=tail -F /opt/gen_logs/logs/access.log

Describe the sink

sdc.sinks.hd.type = hdfs
sdc.sinks.hd.hdfs.path=hdfs://nn01.itversity.com:8020/user/moumita12/flume_demo

sdc.sinks.hd.hdfs.filePrefix=FlumeDemo
sdc.sinks.hd.hdfs.fileSuffix=.txt
sdc.sinks.hd.hdfs.rollInterval=120
sdc.sinks.hd.hdfs.rollsize=1048576
sdc.sinks.hd.hdfs.rollCount=100
sdc.sinks.hd.hdfs.fileType=DataStream

sdc.sinks.spark.type = org.apache.spark.streaming.flume.sink.SparkSink
sdc.sinks.spark.hostname = gw03.itversity.com
sdc.sinks.spark.port = 8123

Use a channel sdcich buffers events in memory

sdc.channels.hdmem.type = memory
sdc.channels.hdmem.capacity = 1000
sdc.channels.hdmem.transactionCapacity = 100

sdc.channels.sparkmem.type = memory
sdc.channels.sparkmem.capacity = 1000
sdc.channels.sparkmem.transactionCapacity = 100

Bind the source and sink to the channel

sdc.sources.ws.channels = hdmem sparkmem
sdc.sinks.hd.channel = hdmem
sdc.sinks.spark.channel = sparkmem

But when i run this, I am getting the below error:-

“2018-09-13 11:59:35,020 main ERROR Unable to create file /var/log/flume-ng/flume.log java.io.IOException: Could not create directory /var/log/flume-ng
at org.apache.logging.log4j.core.util.FileUtils.mkdir(FileUtils.java:127)
at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:144)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:627)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:115)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:144)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:60)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:890)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:513)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:237)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:249)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:122)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:242)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:254)
at org.apache.flume.node.Application.(Application.java:59)”

Could you please have a look? I have attached the screenshot of the

logs.

Thanks,
Moumita