Sqoop import all table issue

Big Data Apache Sqoop
#1

Tried to import all tables to my folder, process went success But the folder didn’t get created. When i tried executing the same code, it errors out saying the folder is exist.

Would you please help me with where I am going wrong?:

[shubhaprasadsamal@gw01 ~]$ sqoop import-all-tables --connect “jdbc:mysql://nn01.itversity.com:3306/retail_db” --username=retail_dba --password=itversity --warehouse-dir=user/shubhaprasadsamal/training/sqoop_import/import_all1
Warning: /usr/hdp/2.5.0.0-1245/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/01/05 14:30:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.0.0-1245
17/01/05 14:30:11 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/01/05 14:30:11 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/01/05 14:30:11 INFO tool.CodeGenTool: Beginning code generation
17/01/05 14:30:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM categories AS t LIMIT 1
17/01/05 14:30:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM categories AS t LIMIT 1
17/01/05 14:30:11 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 14:30:13 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/categories.jar
17/01/05 14:30:13 WARN manager.MySQLManager: It looks like you are importing from mysql.
17/01/05 14:30:13 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
17/01/05 14:30:13 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
17/01/05 14:30:13 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
17/01/05 14:30:13 INFO mapreduce.ImportJobBase: Beginning import of categories
17/01/05 14:30:14 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 14:30:14 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 14:30:14 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 14:30:20 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 14:30:20 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(category_id), MAX(category_id) FROM categories
17/01/05 14:30:20 INFO db.IntegerSplitter: Split size: 14; Num splits: 4 from: 1 to: 58
17/01/05 14:30:21 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 14:30:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5628
17/01/05 14:30:21 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5628
17/01/05 14:30:21 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5628/
17/01/05 14:30:21 INFO mapreduce.Job: Running job: job_1480307771710_5628
17/01/05 14:30:27 INFO mapreduce.Job: Job job_1480307771710_5628 running in uber mode : false
17/01/05 14:30:27 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 14:30:32 INFO mapreduce.Job: map 75% reduce 0%
17/01/05 14:30:33 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 14:30:34 INFO mapreduce.Job: Job job_1480307771710_5628 completed successfully
17/01/05 14:30:35 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642068
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=472
HDFS: Number of bytes written=1029
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=12368
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=12368
Total vcore-milliseconds taken by all map tasks=12368
Total megabyte-milliseconds taken by all map tasks=12664832
Map-Reduce Framework
Map input records=58
Map output records=58
Input split bytes=472
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=206
CPU time spent (ms)=4860
Physical memory (bytes) snapshot=893149184
Virtual memory (bytes) snapshot=11276701696
Total committed heap usage (bytes)=646971392
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=1029
17/01/05 14:30:35 INFO mapreduce.ImportJobBase: Transferred 1.0049 KB in 20.8329 seconds (49.3931 bytes/sec)
17/01/05 14:30:35 INFO mapreduce.ImportJobBase: Retrieved 58 records.
17/01/05 14:30:35 INFO tool.CodeGenTool: Beginning code generation
17/01/05 14:30:35 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM customers AS t LIMIT 1
17/01/05 14:30:35 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/customers.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 14:30:35 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/customers.jar
17/01/05 14:30:35 INFO mapreduce.ImportJobBase: Beginning import of customers
17/01/05 14:30:35 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 14:30:35 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 14:30:35 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 14:30:40 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 14:30:40 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(customer_id), MAX(customer_id) FROM customers
17/01/05 14:30:40 INFO db.IntegerSplitter: Split size: 3108; Num splits: 4 from: 1 to: 12435
17/01/05 14:30:41 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 14:30:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5629
17/01/05 14:30:41 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5629
17/01/05 14:30:41 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5629/
17/01/05 14:30:41 INFO mapreduce.Job: Running job: job_1480307771710_5629
17/01/05 14:30:47 INFO mapreduce.Job: Job job_1480307771710_5629 running in uber mode : false
17/01/05 14:30:47 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 14:30:52 INFO mapreduce.Job: map 50% reduce 0%
17/01/05 14:30:53 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 14:30:54 INFO mapreduce.Job: Job job_1480307771710_5629 completed successfully
17/01/05 14:30:54 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642452
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=487
HDFS: Number of bytes written=953525
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=12226
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=12226
Total vcore-milliseconds taken by all map tasks=12226
Total megabyte-milliseconds taken by all map tasks=12519424
Map-Reduce Framework
Map input records=12435
Map output records=12435
Input split bytes=487
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=234
CPU time spent (ms)=6610
Physical memory (bytes) snapshot=893149184
Virtual memory (bytes) snapshot=11283017728
Total committed heap usage (bytes)=640679936
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=953525
17/01/05 14:30:54 INFO mapreduce.ImportJobBase: Transferred 931.1768 KB in 19.4063 seconds (47.9832 KB/sec)
17/01/05 14:30:54 INFO mapreduce.ImportJobBase: Retrieved 12435 records.
17/01/05 14:30:54 INFO tool.CodeGenTool: Beginning code generation
17/01/05 14:30:54 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM departments AS t LIMIT 1
17/01/05 14:30:54 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/departments.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 14:30:55 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/departments.jar
17/01/05 14:30:55 INFO mapreduce.ImportJobBase: Beginning import of departments
17/01/05 14:30:55 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 14:30:55 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 14:30:55 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 14:31:00 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 14:31:00 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(department_id), MAX(department_id) FROM departments
17/01/05 14:31:00 INFO db.IntegerSplitter: Split size: 24; Num splits: 4 from: 2 to: 100
17/01/05 14:31:01 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 14:31:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5630
17/01/05 14:31:01 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5630
17/01/05 14:31:01 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5630/
17/01/05 14:31:01 INFO mapreduce.Job: Running job: job_1480307771710_5630
17/01/05 14:31:07 INFO mapreduce.Job: Job job_1480307771710_5630 running in uber mode : false
17/01/05 14:31:07 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 14:31:11 INFO mapreduce.Job: map 25% reduce 0%
17/01/05 14:31:12 INFO mapreduce.Job: map 75% reduce 0%
17/01/05 14:31:13 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 14:31:14 INFO mapreduce.Job: Job job_1480307771710_5630 completed successfully
17/01/05 14:31:14 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642008
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=489
HDFS: Number of bytes written=84
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=11658
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=11658
Total vcore-milliseconds taken by all map tasks=11658
Total megabyte-milliseconds taken by all map tasks=11937792
Map-Reduce Framework
Map input records=7
Map output records=7
Input split bytes=489
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=205
CPU time spent (ms)=3690
Physical memory (bytes) snapshot=851218432
Virtual memory (bytes) snapshot=11266453504
Total committed heap usage (bytes)=643301376
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=84
17/01/05 14:31:14 INFO mapreduce.ImportJobBase: Transferred 84 bytes in 19.6204 seconds (4.2813 bytes/sec)
17/01/05 14:31:14 INFO mapreduce.ImportJobBase: Retrieved 7 records.
17/01/05 14:31:14 INFO tool.CodeGenTool: Beginning code generation
17/01/05 14:31:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM order_items AS t LIMIT 1
17/01/05 14:31:14 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/order_items.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 14:31:14 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/order_items.jar
17/01/05 14:31:14 INFO mapreduce.ImportJobBase: Beginning import of order_items
17/01/05 14:31:15 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 14:31:15 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 14:31:15 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 14:31:20 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 14:31:20 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(order_item_id), MAX(order_item_id) FROM order_items
17/01/05 14:31:20 INFO db.IntegerSplitter: Split size: 43049; Num splits: 4 from: 1 to: 172198
17/01/05 14:31:20 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 14:31:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5631
17/01/05 14:31:21 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5631
17/01/05 14:31:21 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5631/
17/01/05 14:31:21 INFO mapreduce.Job: Running job: job_1480307771710_5631
17/01/05 14:31:27 INFO mapreduce.Job: Job job_1480307771710_5631 running in uber mode : false
17/01/05 14:31:27 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 14:31:32 INFO mapreduce.Job: map 75% reduce 0%
17/01/05 14:31:33 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 14:31:34 INFO mapreduce.Job: Job job_1480307771710_5631 completed successfully
17/01/05 14:31:34 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642404
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=512
HDFS: Number of bytes written=5408880
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=13027
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13027
Total vcore-milliseconds taken by all map tasks=13027
Total megabyte-milliseconds taken by all map tasks=13339648
Map-Reduce Framework
Map input records=172198
Map output records=172198
Input split bytes=512
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=237
CPU time spent (ms)=10790
Physical memory (bytes) snapshot=986324992
Virtual memory (bytes) snapshot=11308228608
Total committed heap usage (bytes)=620232704
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=5408880
17/01/05 14:31:34 INFO mapreduce.ImportJobBase: Transferred 5.1583 MB in 19.4125 seconds (272.0988 KB/sec)
17/01/05 14:31:34 INFO mapreduce.ImportJobBase: Retrieved 172198 records.
17/01/05 14:31:34 INFO tool.CodeGenTool: Beginning code generation
17/01/05 14:31:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM orders AS t LIMIT 1
17/01/05 14:31:34 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/orders.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 14:31:34 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/orders.jar
17/01/05 14:31:34 INFO mapreduce.ImportJobBase: Beginning import of orders
17/01/05 14:31:34 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 14:31:34 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 14:31:34 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 14:31:39 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 14:31:39 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(order_id), MAX(order_id) FROM orders
17/01/05 14:31:39 INFO db.IntegerSplitter: Split size: 17220; Num splits: 4 from: 1 to: 68883
17/01/05 14:31:40 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 14:31:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5632
17/01/05 14:31:40 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5632
17/01/05 14:31:40 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5632/
17/01/05 14:31:40 INFO mapreduce.Job: Running job: job_1480307771710_5632
17/01/05 14:31:46 INFO mapreduce.Job: Job job_1480307771710_5632 running in uber mode : false
17/01/05 14:31:46 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 14:31:51 INFO mapreduce.Job: map 50% reduce 0%
17/01/05 14:31:52 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 14:31:53 INFO mapreduce.Job: Job job_1480307771710_5632 completed successfully
17/01/05 14:31:53 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642008
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=469
HDFS: Number of bytes written=2999944
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=13323
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13323
Total vcore-milliseconds taken by all map tasks=13323
Total megabyte-milliseconds taken by all map tasks=13642752
Map-Reduce Framework
Map input records=68883
Map output records=68883
Input split bytes=469
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=244
CPU time spent (ms)=9850
Physical memory (bytes) snapshot=916381696
Virtual memory (bytes) snapshot=11289374720
Total committed heap usage (bytes)=653262848
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=2999944
17/01/05 14:31:54 INFO mapreduce.ImportJobBase: Transferred 2.861 MB in 19.3822 seconds (151.1506 KB/sec)
17/01/05 14:31:54 INFO mapreduce.ImportJobBase: Retrieved 68883 records.
17/01/05 14:31:54 INFO tool.CodeGenTool: Beginning code generation
17/01/05 14:31:54 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM products AS t LIMIT 1
17/01/05 14:31:54 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/products.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 14:31:54 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/5fb47ac3a40e8db9e97de036b2181929/products.jar
17/01/05 14:31:54 INFO mapreduce.ImportJobBase: Beginning import of products
17/01/05 14:31:54 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 14:31:54 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 14:31:54 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 14:31:59 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 14:31:59 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(product_id), MAX(product_id) FROM products
17/01/05 14:31:59 INFO db.IntegerSplitter: Split size: 336; Num splits: 4 from: 1 to: 1345
17/01/05 14:31:59 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 14:31:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5633
17/01/05 14:32:00 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5633
17/01/05 14:32:00 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5633/
17/01/05 14:32:00 INFO mapreduce.Job: Running job: job_1480307771710_5633
17/01/05 14:32:06 INFO mapreduce.Job: Job job_1480307771710_5633 running in uber mode : false
17/01/05 14:32:06 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 14:32:11 INFO mapreduce.Job: map 50% reduce 0%
17/01/05 14:32:12 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 14:32:13 INFO mapreduce.Job: Job job_1480307771710_5633 completed successfully
17/01/05 14:32:13 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642228
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=474
HDFS: Number of bytes written=173993
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=12434
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=12434
Total vcore-milliseconds taken by all map tasks=12434
Total megabyte-milliseconds taken by all map tasks=12732416
Map-Reduce Framework
Map input records=1345
Map output records=1345
Input split bytes=474
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=240
CPU time spent (ms)=5600
Physical memory (bytes) snapshot=903610368
Virtual memory (bytes) snapshot=11285983232
Total committed heap usage (bytes)=644874240
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=173993
17/01/05 14:32:13 INFO mapreduce.ImportJobBase: Transferred 169.915 KB in 19.1079 seconds (8.8924 KB/sec)
17/01/05 14:32:13 INFO mapreduce.ImportJobBase: Retrieved 1345 records.
[shubhaprasadsamal@gw01 ~]$ hadoop fs -ls /user/shubhaprasadsamal/training/sqoop_import
Found 5 items
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 19:21 /user/shubhaprasadsamal/training/sqoop_import/product-avro
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 19:26 /user/shubhaprasadsamal/training/sqoop_import/product-avro1
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 20:02 /user/shubhaprasadsamal/training/sqoop_import/product_seq
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 20:08 /user/shubhaprasadsamal/training/sqoop_import/product_text
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 19:09 /user/shubhaprasadsamal/training/sqoop_import/products
[shubhaprasadsamal@gw01 ~]$ sqoop import-all-tables --connect “jdbc:mysql://nn01.itversity.com:3306/retail_db” --username=retail_dba --password=itversity --warehouse-dir=user/shubhaprasadsamal/training/sqoop_import/import_all1
Warning: /usr/hdp/2.5.0.0-1245/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/01/05 14:34:22 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.0.0-1245
17/01/05 14:34:22 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/01/05 14:34:23 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/01/05 14:34:23 INFO tool.CodeGenTool: Beginning code generation
17/01/05 14:34:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM categories AS t LIMIT 1
17/01/05 14:34:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM categories AS t LIMIT 1
17/01/05 14:34:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/e647904e2b8fbcf0b058b327ee4fd675/categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 14:34:24 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/e647904e2b8fbcf0b058b327ee4fd675/categories.jar
17/01/05 14:34:24 WARN manager.MySQLManager: It looks like you are importing from mysql.
17/01/05 14:34:24 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
17/01/05 14:34:24 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
17/01/05 14:34:24 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
17/01/05 14:34:24 INFO mapreduce.ImportJobBase: Beginning import of categories
17/01/05 14:34:25 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 14:34:26 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 14:34:26 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 14:34:26 ERROR tool.ImportAllTablesTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://nn01.itversity.com:8020/user/shubhaprasadsamal/user/shubhaprasadsamal/training/sqoop_import/import_all1/categories already exists

#2

17/01/05 14:34:26 ERROR tool.ImportAllTablesTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://nn01.itversity.com:8020/user/shubhaprasadsamal/user/shubhaprasadsamal/training/sqoop_import/import_all1/categories already exists

the dir already exists. Remove it and execute it again

#3

Thanks Raghav for looking into the issue:

as you suggested I remove the directories and add a new directory which is not exist i.e

sqoop import-all-tables --connect “jdbc:mysql://nn01.itversity.com:3306/retail_db” --username=retail_dba --password=itversity --warehouse-dir=user/shubhaprasadsamal/training/sqoop_import/import_all2

and executed it. went to success, but I still didn’t find the new directory by “fs -ls”

and when I ran the same code again, it threw me “folder exist error” again:

Please look at the code in sequential manner:

Step-1:

[shubhaprasadsamal@gw01 ~]$ sqoop import-all-tables --connect “jdbc:mysql://nn01.itversity.com:3306/retail_db” --username=retail_dba --password=itversity --warehouse-dir=user/shubhaprasadsamal/training/sqoop_import/import_all2
Warning: /usr/hdp/2.5.0.0-1245/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/01/05 15:14:55 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.0.0-1245
17/01/05 15:14:55 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/01/05 15:14:55 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/01/05 15:14:55 INFO tool.CodeGenTool: Beginning code generation
17/01/05 15:14:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM categories AS t LIMIT 1
17/01/05 15:14:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM categories AS t LIMIT 1
17/01/05 15:14:55 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 15:14:57 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/categories.jar
17/01/05 15:14:57 WARN manager.MySQLManager: It looks like you are importing from mysql.
17/01/05 15:14:57 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
17/01/05 15:14:57 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
17/01/05 15:14:57 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
17/01/05 15:14:57 INFO mapreduce.ImportJobBase: Beginning import of categories
17/01/05 15:14:58 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 15:14:58 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 15:14:58 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 15:15:05 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 15:15:05 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(category_id), MAX(category_id) FROM categories
17/01/05 15:15:05 INFO db.IntegerSplitter: Split size: 14; Num splits: 4 from: 1 to: 58
17/01/05 15:15:05 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 15:15:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5640
17/01/05 15:15:06 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5640
17/01/05 15:15:06 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5640/
17/01/05 15:15:06 INFO mapreduce.Job: Running job: job_1480307771710_5640
17/01/05 15:15:12 INFO mapreduce.Job: Job job_1480307771710_5640 running in uber mode : false
17/01/05 15:15:12 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 15:15:17 INFO mapreduce.Job: map 75% reduce 0%
17/01/05 15:15:18 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 15:15:18 INFO mapreduce.Job: Job job_1480307771710_5640 completed successfully
17/01/05 15:15:18 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642068
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=472
HDFS: Number of bytes written=1029
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=12474
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=12474
Total vcore-milliseconds taken by all map tasks=12474
Total megabyte-milliseconds taken by all map tasks=12773376
Map-Reduce Framework
Map input records=58
Map output records=58
Input split bytes=472
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=241
CPU time spent (ms)=5480
Physical memory (bytes) snapshot=911945728
Virtual memory (bytes) snapshot=11284697088
Total committed heap usage (bytes)=652738560
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=1029
17/01/05 15:15:18 INFO mapreduce.ImportJobBase: Transferred 1.0049 KB in 20.6236 seconds (49.8942 bytes/sec)
17/01/05 15:15:18 INFO mapreduce.ImportJobBase: Retrieved 58 records.
17/01/05 15:15:18 INFO tool.CodeGenTool: Beginning code generation
17/01/05 15:15:18 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM customers AS t LIMIT 1
17/01/05 15:15:18 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/customers.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 15:15:19 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/customers.jar
17/01/05 15:15:19 INFO mapreduce.ImportJobBase: Beginning import of customers
17/01/05 15:15:19 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 15:15:19 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 15:15:19 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 15:15:24 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 15:15:24 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(customer_id), MAX(customer_id) FROM customers
17/01/05 15:15:24 INFO db.IntegerSplitter: Split size: 3108; Num splits: 4 from: 1 to: 12435
17/01/05 15:15:25 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 15:15:25 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5641
17/01/05 15:15:25 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5641
17/01/05 15:15:25 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5641/
17/01/05 15:15:25 INFO mapreduce.Job: Running job: job_1480307771710_5641
17/01/05 15:15:31 INFO mapreduce.Job: Job job_1480307771710_5641 running in uber mode : false
17/01/05 15:15:31 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 15:15:35 INFO mapreduce.Job: map 25% reduce 0%
17/01/05 15:15:36 INFO mapreduce.Job: map 75% reduce 0%
17/01/05 15:15:37 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 15:15:38 INFO mapreduce.Job: Job job_1480307771710_5641 completed successfully
17/01/05 15:15:38 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642452
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=487
HDFS: Number of bytes written=953525
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=12299
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=12299
Total vcore-milliseconds taken by all map tasks=12299
Total megabyte-milliseconds taken by all map tasks=12594176
Map-Reduce Framework
Map input records=12435
Map output records=12435
Input split bytes=487
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=246
CPU time spent (ms)=5910
Physical memory (bytes) snapshot=862474240
Virtual memory (bytes) snapshot=11275591680
Total committed heap usage (bytes)=637009920
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=953525
17/01/05 15:15:38 INFO mapreduce.ImportJobBase: Transferred 931.1768 KB in 19.6525 seconds (47.382 KB/sec)
17/01/05 15:15:38 INFO mapreduce.ImportJobBase: Retrieved 12435 records.
17/01/05 15:15:38 INFO tool.CodeGenTool: Beginning code generation
17/01/05 15:15:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM departments AS t LIMIT 1
17/01/05 15:15:38 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/departments.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 15:15:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/departments.jar
17/01/05 15:15:39 INFO mapreduce.ImportJobBase: Beginning import of departments
17/01/05 15:15:39 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 15:15:39 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 15:15:39 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 15:15:45 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 15:15:45 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(department_id), MAX(department_id) FROM departments
17/01/05 15:15:45 INFO db.IntegerSplitter: Split size: 24; Num splits: 4 from: 2 to: 100
17/01/05 15:15:45 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 15:15:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5642
17/01/05 15:15:45 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5642
17/01/05 15:15:45 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5642/
17/01/05 15:15:45 INFO mapreduce.Job: Running job: job_1480307771710_5642
17/01/05 15:15:51 INFO mapreduce.Job: Job job_1480307771710_5642 running in uber mode : false
17/01/05 15:15:51 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 15:15:56 INFO mapreduce.Job: map 75% reduce 0%
17/01/05 15:15:57 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 15:15:59 INFO mapreduce.Job: Job job_1480307771710_5642 completed successfully
17/01/05 15:15:59 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642008
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=489
HDFS: Number of bytes written=84
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=11676
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=11676
Total vcore-milliseconds taken by all map tasks=11676
Total megabyte-milliseconds taken by all map tasks=11956224
Map-Reduce Framework
Map input records=7
Map output records=7
Input split bytes=489
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=196
CPU time spent (ms)=3680
Physical memory (bytes) snapshot=857882624
Virtual memory (bytes) snapshot=11264659456
Total committed heap usage (bytes)=651689984
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=84
17/01/05 15:15:59 INFO mapreduce.ImportJobBase: Transferred 84 bytes in 19.8663 seconds (4.2283 bytes/sec)
17/01/05 15:15:59 INFO mapreduce.ImportJobBase: Retrieved 7 records.
17/01/05 15:15:59 INFO tool.CodeGenTool: Beginning code generation
17/01/05 15:15:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM order_items AS t LIMIT 1
17/01/05 15:15:59 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/order_items.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 15:15:59 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/order_items.jar
17/01/05 15:15:59 INFO mapreduce.ImportJobBase: Beginning import of order_items
17/01/05 15:15:59 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 15:15:59 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 15:15:59 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 15:16:05 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 15:16:05 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(order_item_id), MAX(order_item_id) FROM order_items
17/01/05 15:16:05 INFO db.IntegerSplitter: Split size: 43049; Num splits: 4 from: 1 to: 172198
17/01/05 15:16:05 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 15:16:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5643
17/01/05 15:16:06 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5643
17/01/05 15:16:06 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5643/
17/01/05 15:16:06 INFO mapreduce.Job: Running job: job_1480307771710_5643
17/01/05 15:16:12 INFO mapreduce.Job: Job job_1480307771710_5643 running in uber mode : false
17/01/05 15:16:12 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 15:16:17 INFO mapreduce.Job: map 50% reduce 0%
17/01/05 15:16:18 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 15:16:19 INFO mapreduce.Job: Job job_1480307771710_5643 completed successfully
17/01/05 15:16:19 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642404
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=512
HDFS: Number of bytes written=5408880
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=14090
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=14090
Total vcore-milliseconds taken by all map tasks=14090
Total megabyte-milliseconds taken by all map tasks=14428160
Map-Reduce Framework
Map input records=172198
Map output records=172198
Input split bytes=512
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=201
CPU time spent (ms)=10840
Physical memory (bytes) snapshot=970285056
Virtual memory (bytes) snapshot=11314323456
Total committed heap usage (bytes)=622854144
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=5408880
17/01/05 15:16:19 INFO mapreduce.ImportJobBase: Transferred 5.1583 MB in 20.2677 seconds (260.617 KB/sec)
17/01/05 15:16:19 INFO mapreduce.ImportJobBase: Retrieved 172198 records.
17/01/05 15:16:19 INFO tool.CodeGenTool: Beginning code generation
17/01/05 15:16:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM orders AS t LIMIT 1
17/01/05 15:16:19 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/orders.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 15:16:19 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/orders.jar
17/01/05 15:16:19 INFO mapreduce.ImportJobBase: Beginning import of orders
17/01/05 15:16:19 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 15:16:19 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 15:16:19 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 15:16:25 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 15:16:25 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(order_id), MAX(order_id) FROM orders
17/01/05 15:16:25 INFO db.IntegerSplitter: Split size: 17220; Num splits: 4 from: 1 to: 68883
17/01/05 15:16:25 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 15:16:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5644
17/01/05 15:16:26 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5644
17/01/05 15:16:26 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5644/
17/01/05 15:16:26 INFO mapreduce.Job: Running job: job_1480307771710_5644
17/01/05 15:16:32 INFO mapreduce.Job: Job job_1480307771710_5644 running in uber mode : false
17/01/05 15:16:32 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 15:16:37 INFO mapreduce.Job: map 50% reduce 0%
17/01/05 15:16:38 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 15:16:39 INFO mapreduce.Job: Job job_1480307771710_5644 completed successfully
17/01/05 15:16:39 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642008
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=469
HDFS: Number of bytes written=2999944
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=13327
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13327
Total vcore-milliseconds taken by all map tasks=13327
Total megabyte-milliseconds taken by all map tasks=13646848
Map-Reduce Framework
Map input records=68883
Map output records=68883
Input split bytes=469
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=243
CPU time spent (ms)=9950
Physical memory (bytes) snapshot=948129792
Virtual memory (bytes) snapshot=11306110976
Total committed heap usage (bytes)=636485632
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=2999944
17/01/05 15:16:39 INFO mapreduce.ImportJobBase: Transferred 2.861 MB in 19.8369 seconds (147.6858 KB/sec)
17/01/05 15:16:39 INFO mapreduce.ImportJobBase: Retrieved 68883 records.
17/01/05 15:16:39 INFO tool.CodeGenTool: Beginning code generation
17/01/05 15:16:39 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM products AS t LIMIT 1
17/01/05 15:16:39 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/products.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 15:16:39 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/39d6bb4de47d5223db260429bd491766/products.jar
17/01/05 15:16:39 INFO mapreduce.ImportJobBase: Beginning import of products
17/01/05 15:16:39 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 15:16:39 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 15:16:39 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 15:16:45 INFO db.DBInputFormat: Using read commited transaction isolation
17/01/05 15:16:45 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(product_id), MAX(product_id) FROM products
17/01/05 15:16:45 INFO db.IntegerSplitter: Split size: 336; Num splits: 4 from: 1 to: 1345
17/01/05 15:16:45 INFO mapreduce.JobSubmitter: number of splits:4
17/01/05 15:16:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1480307771710_5645
17/01/05 15:16:46 INFO impl.YarnClientImpl: Submitted application application_1480307771710_5645
17/01/05 15:16:46 INFO mapreduce.Job: The url to track the job: http://rm01.itversity.com:8088/proxy/application_1480307771710_5645/
17/01/05 15:16:46 INFO mapreduce.Job: Running job: job_1480307771710_5645
17/01/05 15:16:52 INFO mapreduce.Job: Job job_1480307771710_5645 running in uber mode : false
17/01/05 15:16:52 INFO mapreduce.Job: map 0% reduce 0%
17/01/05 15:16:57 INFO mapreduce.Job: map 50% reduce 0%
17/01/05 15:16:58 INFO mapreduce.Job: map 100% reduce 0%
17/01/05 15:16:59 INFO mapreduce.Job: Job job_1480307771710_5645 completed successfully
17/01/05 15:16:59 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=642228
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=474
HDFS: Number of bytes written=173993
HDFS: Number of read operations=16
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
Job Counters
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=12646
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=12646
Total vcore-milliseconds taken by all map tasks=12646
Total megabyte-milliseconds taken by all map tasks=12949504
Map-Reduce Framework
Map input records=1345
Map output records=1345
Input split bytes=474
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=192
CPU time spent (ms)=5620
Physical memory (bytes) snapshot=914550784
Virtual memory (bytes) snapshot=11295805440
Total committed heap usage (bytes)=666370048
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=173993
17/01/05 15:16:59 INFO mapreduce.ImportJobBase: Transferred 169.915 KB in 19.2939 seconds (8.8067 KB/sec)
17/01/05 15:16:59 INFO mapreduce.ImportJobBase: Retrieved 1345 records.

STEP:2

[shubhaprasadsamal@gw01 ~]$ hadoop fs -ls /user/shubhaprasadsamal/training/sqoop_import
Found 5 items
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 19:21 /user/shubhaprasadsamal/training/sqoop_import/product-avro
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 19:26 /user/shubhaprasadsamal/training/sqoop_import/product-avro1
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 20:02 /user/shubhaprasadsamal/training/sqoop_import/product_seq
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 20:08 /user/shubhaprasadsamal/training/sqoop_import/product_text
drwxr-xr-x - shubhaprasadsamal hdfs 0 2017-01-04 19:09 /user/shubhaprasadsamal/training/sqoop_import/products

STEP:3

[shubhaprasadsamal@gw01 ~]$ sqoop import-all-tables --connect “jdbc:mysql://nn01.itversity.com:3306/retail_db” --username=retail_dba --password=itversity --warehouse-dir=user/shubhaprasadsamal/training/sqoop_import/import_all2
Warning: /usr/hdp/2.5.0.0-1245/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/01/05 15:19:18 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.0.0-1245
17/01/05 15:19:18 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/01/05 15:19:19 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/01/05 15:19:19 INFO tool.CodeGenTool: Beginning code generation
17/01/05 15:19:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM categories AS t LIMIT 1
17/01/05 15:19:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM categories AS t LIMIT 1
17/01/05 15:19:19 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce
Note: /tmp/sqoop-shubhaprasadsamal/compile/cdd8cf8c356f90c46f6409dfbf9f8a16/categories.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/05 15:19:20 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-shubhaprasadsamal/compile/cdd8cf8c356f90c46f6409dfbf9f8a16/categories.jar
17/01/05 15:19:20 WARN manager.MySQLManager: It looks like you are importing from mysql.
17/01/05 15:19:20 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
17/01/05 15:19:20 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
17/01/05 15:19:20 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
17/01/05 15:19:20 INFO mapreduce.ImportJobBase: Beginning import of categories
17/01/05 15:19:22 INFO impl.TimelineClientImpl: Timeline service address: http://rm01.itversity.com:8188/ws/v1/timeline/
17/01/05 15:19:22 INFO client.RMProxy: Connecting to ResourceManager at rm01.itversity.com/172.16.1.106:8050
17/01/05 15:19:22 INFO client.AHSProxy: Connecting to Application History server at rm01.itversity.com/172.16.1.106:10200
17/01/05 15:19:22 ERROR tool.ImportAllTablesTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://nn01.itversity.com:8020/user/shubhaprasadsamal/user/shubhaprasadsamal/training/sqoop_import/import_all2/categories already exists

Any guidance will be deeply appreciated.

Thanks,
Shitansu

#4

please give the warehouse-dir path correctly. There is a “/” missing.
Please check the dirs under /user/shubhaprasadsamal

u will be able to find the dirs.

Please paste the error rather than whole log :slight_smile:

#5

Thanks Raghav. It’s resolved.