Big Data Development Lifecycle


Hello guys,

I am asking this question to the people currently working in the industry on Big Data. Can you guys explain the how the development goes in the company? I guess there are two types of roles in big data development.

  1. As an application developer who mainly uses Java API’s to write applications and these applications perform the required extraction, transformation, and loading to SQL or NoSQL databases.
  2. As an ETL developer who uses Hive, SparkSQL and does the database level design, extraction, transformations, and loading to SQL or NoSQL databases.

Correct me if im wrong with my understanding of the responsibilities of the above two roles. And please explain the development lifecycle involving big data. Would really appreciate if someone could take time and explain and I guess this is a major question to many recent graduates coming into big data.

Thank you