Data Ingest - real time, near real time and streaming analytics - Kafka and Spark Streaming – Develop and build application

Integration – Kafka and Spark Streaming

  • Import necessary packages
  • Create set for a list of topics and dict for Kafka broker list and other
    parameters
  • Create input stream using relevant APIs by passing topics list and
    broker properties
  • Process the data using relevant DStream APIs
  • Download dependencies on the gateway node
  • Ship the code and run on the cluster including Kafka dependencies

Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster