Not able to read a topic in Kafka using Pyspark

Hi,
I have created a topic in Kafka which I am able to send and receive as a producer/consumer in console. I can see that.
[vikct001@gw02 kafka]$ sh kafka_producer.sh
>My Name is Vikrant Singh Rana

I then started my spark session as:
pyspark2 --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0

df = spark \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", "nn01.itversity.com:2181,nn02.itversity.com:2181,rm01.itversity.com:2181") \
.option("startingOffsets","earliest") \
.option("subscribe", "testing_vikrant_kafka") \
.load()

df.writeStream \
.format("console") \
.option("truncate","false") \
.start() \
.awaitTermination()

it’s giving me below set of repeating commands

19/09/01 19:40:59 WARN NetworkClient: Bootstrap broker nn02.itversity.com:2181 disconnected
19/09/01 19:40:59 WARN NetworkClient: Bootstrap broker rm01.itversity.com:2181 disconnected
19/09/01 19:40:59 WARN NetworkClient: Bootstrap broker nn02.itversity.com:2181 disconnected
19/09/01 19:40:59 WARN NetworkClient: Bootstrap broker rm01.itversity.com:2181 disconnected
19/09/01 19:40:59 WARN NetworkClient: Bootstrap broker nn01.itversity.com:2181 disconnected
19/09/01 19:40:59 WARN NetworkClient: Bootstrap broker nn02.itversity.com:2181 disconnected

Kindly help

@Vikrant_Singh_Rana

The port number that you mentioned for bootstrap servers is actually zookeeper port. Instead of that you need to mention the port number of Kafka brokers in order to reach the Kafka topics.

1 Like

Yeah. This helped. Thanks