Kafka connect is failing while trying to flush the message to consumer


#1

Getting below errors while trying to explore kafka connect feature to import and export data from a file.

connect-standalone.sh hadoop_config/connect-standalone.properties hadoop_config/connect-file-source.properties hadoop_config/connect-file-sink.properties

ERRORS in logs as follows…

[2018-07-05 08:30:42,050] INFO Kafka version : 0.10.0.2.5.0.0-1245 (org.apache.kafka.common.utils.AppInfoParser:83)
[2018-07-05 08:30:42,050] INFO Kafka commitId : dae559f56f07e2cd (org.apache.kafka.common.utils.AppInfoParser:84)
[2018-07-05 08:30:42,051] INFO Created connector local-file-sink (org.apache.kafka.connect.cli.ConnectStandalone:91)
[2018-07-05 08:30:42,060] INFO Sink task WorkerSinkTask{id=local-file-sink-0} finished initialization and start (org.apache.kafka.connect.runtime.WorkerSinkTask:208)
[2018-07-05 08:30:55,833] ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.conne
ct.runtime.WorkerSourceTask:291)
[2018-07-05 08:30:55,833] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:109)
[2018-07-05 08:31:10,834] ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.conne
ct.runtime.WorkerSourceTask:291)
[2018-07-05 08:31:10,834] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:109)
[2018-07-05 08:31:25,834] ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.conne
ct.runtime.WorkerSourceTask:291)
[2018-07-05 08:31:25,835] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:109)
[2018-07-05 08:31:40,835] ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.conne
ct.runtime.WorkerSourceTask:291)
[2018-07-05 08:31:40,835] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:109)
[2018-07-05 08:31:55,836] ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.conne
ct.runtime.WorkerSourceTask:291)
[2018-07-05 08:31:55,836] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:109)
[2018-07-05 08:32:10,836] ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.conne
ct.runtime.WorkerSourceTask:291)
[2018-07-05 08:32:10,837] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:109)
[2018-07-05 08:32:25,837] ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.conne
ct.runtime.WorkerSourceTask:291)
[2018-07-05 08:32:25,837] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:109)
[2018-07-05 08:32:40,838] ERROR Failed to flush WorkerSourceTask{id=local-file-source-0}, timed out while waiting for producer to flush outstanding 1 messages (org.apache.kafka.conne
ct.runtime.WorkerSourceTask:291)
[2018-07-05 08:32:40,838] ERROR Failed to commit offsets for WorkerSourceTask{id=local-file-source-0} (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter:109)


#2

Any update on this issue?


#3

This is not good… no onefrom admin ( Bigdata Labs)
responding to my question here even after 11 + hours.


#4

@deepak_kotla - sorry for being late.

We do not have Kafka connect installed. It is not typical lab issue where the services are down. Also the issue is not categorised under bigdata-labs. Only issues categorised under bigdata-labs is taken care by support team otherwise it is left to community.


#5

What is your lab id? I tried deepak_kotla but it is not there. You need to provide more information so that we can have a look at the issue.


#6

my lab id is djsphark1.

I tried put in bigdata-labs but, i don’t see any category related to Kafka.

Thanks,
Deepak