Sqoop import (warehouse-dir command hanging)


#1

[jayshawusa@gw01 ~]$ hadoop fs -ls -R /user/jayshawusa/sqoop_import/retail_db
ls: `/user/jayshawusa/sqoop_import/retail_db’: No such file or directory

*[jayshawusa@gw01 ~]$ sqoop import *
*> --connect jdbc:mysql://ms.itversity.com:3306/retail_db *
*> --username retail_user *
> --password itversity *
> --table order_items *
> --warehouse-dir /user/jayshawusa/sqoop_import/retail_db
Warning: /usr/hdp/2.5.0.0-1245/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
18/01/14 09:46:24 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.0.0-1245
18/01/14 09:46:24 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/01/14 09:46:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/01/14 09:46:24 INFO tool.CodeGenTool: Beginning code generation
18/01/14 09:46:25 INFO manager.SqlManager: Executing SQL statement: SELECT t.
FROM order_items AS t LIMIT 1
18/01/14 09:46:25 INFO manager.SqlManager: Executing SQL statement: SELECT t.
FROM order_items AS t LIMIT 1
18/01/14 09:46:25 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce

18/01/14 09:46:39 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(order_item_id), MAX(order_item_id) FROM order_items
18/01/14 09:46:39 INFO db.IntegerSplitter: Split size: 43049; Num splits: 4 from: 1 to: 172198
18/01/14 09:46:39 INFO mapreduce.JobSubmitter: number of splits:4
18/01/14 09:46:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1507687444776_38757
18/01/14 09:46:41 INFO impl.YarnClientImpl: Submitted application application_1507687444776_38757
18/01/14 09:46:41 INFO mapreduce.Job: The url to track the job:
18/01/14 09:46:41 INFO mapreduce.Job: Running job: job_1507687444776_38757


#2

@jayshawusa

Not sure sqoop import * works but please try to use sqoop import-all-tables like below and confirm:

sqoop import-all-tables
–connect jdbc:mysql://ms.itversity.com:3306/retail_db
–username retail_user
–password itversity
–warehouse-dir /user/<YOUR_USER_ID>/sqoop_import/retail_db
–autoreset-to-one-mapper

Thanks
Venkat


#3

I am following Durga udemy video… it is hanging.
When I checked online status. . some capacity issue on your side.
Could you please clear the load on your side and confirm. Thanks.

[Sun Jan 14 10:17:43 -0500 2018] Application is added to the scheduler and is not yet activated. Queue’s AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:2048, vCores:1>; Queue Resource Limit for AM = <memory:24576, vCores:1>; User AM Resource Limit of the queue = <memory:24576, vCores:1>; Queue AM Resource Usage = <memory:24576, vCores:19>;
Unmanaged Application: false


#4

Also your command of “import-all-tables” hanging too
it is clearly capacity issue on itversity side
Please check this URL
http://rm01.itversity.com:8088/proxy/application_1507687444776_38769/


#5

@jayshawusa:
Hold on. I reproduced your issue. Seems, system is under maintenance. Probably it will resolve soon.
Will keep you posted.
Thanks
Venkat