Can SPARK-SCALA be used as alternative to SQOOP to transfer data from RDBMS to HDFS / HIVE?

Can SPARK-SCALA be used as alternative to SQOOP to transfer data from RDBMS to HDFS / HIVE?

Any limitations ? Assuming SQOOP handles batch loads for high volumes of data?? Can spark achieve the same?

Thoughts please- what are the limitations and why only one approach is better suitable than other?

This question was asked in an interview.
yes, we can.
I found the below limitations. these are just my thoughts:

@itversity can u plz clarify?

  1. There is a password encryption class in sqoop using which you can encode the db password. I did not find any such provision in spark
  2. sqoop framework will take care of incremental loading by storing the prev run details in metastore. We have to have our own such process in spark
1 Like