Kafka message queue design - Project


#1

I am building a dashboard with metrics from a live chat stream data. But, I am not able to come up with a satisfactory design. As for now, I receive live chat comments on different video channels from a website twitch.tv. My problem here is coming up with a high level design for Kafka and Spark-Streaming. I have implemented message queue such that I have new topic for each channel.

But, I am not sure if that is a correct way of doing things. I have spark consumers doing basic aggregations on the data. What would be the recommended way of doing things if I were to use Kafka? Thank you.