Incremental Export to SQL Server using Sqoop

Incremental Export to SQL Server using Sqoop
0.0 0

#1

How to perform Incremental Exports to SQL server using Sqoop.
i have data till 15th of a month in SQL server and I want to move the rest 15 days of data from my Hive tables to SQL Server.
How can we perform that?


#2

I would say try to add a date column to your hive table. Adding date or timestamp column to any table helps with incremental imports and exports. In your case, once you are done adding the date column. You can select the data after 15th of a month by specifying the exact date in the where clause from which you want the data to be exported.

I am not sure if we have a sqoop argument for incremental export but try to explore “update-key” argument. If you don’t find any predefined argument go with the above process. Usually, in most of the real-time projects from my experience, we add timestamp column for most of the hive tables to perform such actions.


#3

Well I used --update-mode allowinsert clause and it worked.