Unable to push/pull message to kafka topic in lab from IDE installed local

#1

Dear Team,

Am unable to publish the message to kafka topic outside lab through IDE installed locally.
settings:
Bootstrap-server: wn01.itversity.com:6667

Below is my program

import java.util.{Date, Properties}

import org.apache.kafka.common.serialization.{StringSerializer, StringDeserializer}
import org.apache.kafka.common._
import org.apache.kafka.clients._
import org.apache.kafka.clients.producer.{KafkaProducer, ProducerConfig, ProducerRecord}

object kafkaProducer {
def main(args: Array[String]): Unit = {

val bootStarpServer = “wn01.itversity.com:6667
//val bootStarpServer = “localhost:9092”

val props = new Properties()
// props.put(“bootstrap.servers”,bootStarpServer)
// props.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”)
// props.put(“value.serializer”, “org.apache.kafka.common.serialization.StringSerializer”)

// //alternate method
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootStarpServer)
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, “org.apache.kafka.common.serialization.StringSerializer”)
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, “org.apache.kafka.common.serialization.StringSerializer”)

val producer = new KafkaProducerString,String

val producerRecord = new ProducerRecordString,String

producer.send(producerRecord)

producer.flush()
producer.close()

}

}

Is any addition credentials need to add in local, let me know please.

Error:
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 2.1.1
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : 21234bee31165527
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (wn01.itversity.com/149.56.240.101:6667) could not be established. Broker may not be available.
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (wn01.itversity.com/149.56.240.101:6667) could not be established. Broker may not be available.
[main] INFO org.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

@itversity @vinodnerella @itversity1

0 Likes

#2

Please follow the github link to set up and run the kafka example

0 Likes

#3

Hi I have imported this project and ran . nothing happens after producer.send. no message in consumer console in itversity. Can you please help?

import java.util.Properties
import com.typesafe.config.ConfigFactory
import org.apache.kafka.clients.producer.{KafkaProducer, ProducerConfig, ProducerRecord, RecordMetadata}

object KafkaProducerExample {

  def main(args: Array[String]): Unit = {

    val conf = ConfigFactory.load(this.getClass().getClassLoader())
    val envConfig = conf.getConfig(args(0))
    println(envConfig)
    val props = new Properties()
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, envConfig.getString("bootstrap.server"))
    props.put(ProducerConfig.CLIENT_ID_CONFIG, "KafkaProducerExample")
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")
    val producer = new KafkaProducer[String, String](props)
    println("producer created")

    val data = new ProducerRecord[String, String](envConfig.getString("topic"), "hello", "Universe")
    println(data.topic()+"\n"+data.partition()+"\n"+data.toString)
     producer.send(data)
    println("sent msg")
      producer.close()
    println("closed producer")
  }
}

Preformatted text

Message is sent but when i check in consumer console in lab, it isn’t there.
is there something i have missed in configuration? Please help
I can’t see consumer receiving the messages either in lab console or in intellij IDEA.
@itversity @venkateshm @dgadiraju @vinodnerella @Sunil_Itversity

0 Likes

#4

We have not exposed wn01.itversity.com to the outside world. You will not be able to run using IDE.

You have to build the jar and deploy on the cluster.

0 Likes

#5

Ok. But producer is working fine.
And also I am trying to learn Kafka and Spark structured streaming with scala, is there any course that will be helpful for me. I saw one Durga sir video over 1 hour duration but it was in pyspark. I willing to join the course if there is any scala alternative. Also will it include any project experience?

0 Likes

#6

Can you share your complete code base in GitHub?

0 Likes

#7

I have not pushed complete code.will share in the evening. but it is same as Durga sir code KafkaProducerExample and KafkaConsumerExample. I have just changed my topic name in properties file

0 Likes

#8

What is the command you are running to consume the messages? Please produce the messages once the consumer is started and see if it working.

0 Likes