Sqoop export failure -- help needed

sqoop
Sqoop export failure -- help needed
5.0 1

#1

Hi Durga Sir,
I created a device_export table and accounts_export table on the database that I want to upload data into thru mysql. I tried sqoop export command on my virtual machine that I got from cloudera. But I only get error messages. I am attaching my queries and also the screen output here. Can you please help?

sqoop export --connect jdbc:mysql://localhost/loudacre --username training --password training --table accounts_export --export-dir /user/hive/warehouse/sqoop_import.db/accounts \

–num-mappers 2
17/05/17 14:17:48 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
17/05/17 14:17:48 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/05/17 14:17:48 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/05/17 14:17:48 INFO tool.CodeGenTool: Beginning code generation
17/05/17 14:17:49 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM accounts_export AS t LIMIT 1
17/05/17 14:17:49 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM accounts_export AS t LIMIT 1
17/05/17 14:17:49 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-training/compile/d3d89c620e7e99ad02bed228456a18b9/accounts_export.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/05/17 14:17:52 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-training/compile/d3d89c620e7e99ad02bed228456a18b9/accounts_export.jar
17/05/17 14:17:52 INFO mapreduce.ExportJobBase: Beginning export of accounts_export
17/05/17 14:17:52 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
17/05/17 14:17:52 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/05/17 14:17:52 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts
17/05/17 14:17:54 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
17/05/17 14:17:54 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
17/05/17 14:17:54 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/05/17 14:17:54 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/17 14:17:57 INFO input.FileInputFormat: Total input paths to process : 1
17/05/17 14:17:57 INFO input.FileInputFormat: Total input paths to process : 1
17/05/17 14:17:57 INFO mapreduce.JobSubmitter: number of splits:2
17/05/17 14:17:57 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
17/05/17 14:17:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1495052924480_0003
17/05/17 14:17:57 INFO impl.YarnClientImpl: Submitted application application_1495052924480_0003
17/05/17 14:17:57 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1495052924480_0003/
17/05/17 14:17:57 INFO mapreduce.Job: Running job: job_1495052924480_0003
17/05/17 14:18:09 INFO mapreduce.Job: Job job_1495052924480_0003 running in uber mode : false
17/05/17 14:18:09 INFO mapreduce.Job: map 0% reduce 0%
17/05/17 14:18:17 INFO mapreduce.Job: map 100% reduce 0%
17/05/17 14:18:18 INFO mapreduce.Job: Job job_1495052924480_0003 failed with state FAILED due to: Task failed task_1495052924480_0003_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

17/05/17 14:18:18 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=1
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=5966
Total vcore-seconds taken by all map tasks=5966
Total megabyte-seconds taken by all map tasks=1527296
17/05/17 14:18:18 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
17/05/17 14:18:18 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 23.697 seconds (0 bytes/sec)
17/05/17 14:18:18 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
17/05/17 14:18:18 INFO mapreduce.ExportJobBase: Exported 0 records.
17/05/17 14:18:18 ERROR tool.ExportTool: Error during export: Export job failed!
[training@localhost ~]$ sqoop export --connect jdbc:mysql://localhost/loudacre --username training --password training --table accounts_export --export-dir /user/hive/warehouse/sqoop_import.db/accounts --num-mappers 4
17/05/17 14:27:14 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
17/05/17 14:27:14 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/05/17 14:27:14 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/05/17 14:27:14 INFO tool.CodeGenTool: Beginning code generation
17/05/17 14:27:15 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM accounts_export AS t LIMIT 1
17/05/17 14:27:15 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM accounts_export AS t LIMIT 1
17/05/17 14:27:15 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-training/compile/21dcf53ad30c5ce70d4fdf77ca83fb49/accounts_export.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/05/17 14:27:18 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-training/compile/21dcf53ad30c5ce70d4fdf77ca83fb49/accounts_export.jar
17/05/17 14:27:18 INFO mapreduce.ExportJobBase: Beginning export of accounts_export
17/05/17 14:27:18 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
17/05/17 14:27:19 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/05/17 14:27:19 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts
17/05/17 14:27:20 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
17/05/17 14:27:20 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
17/05/17 14:27:20 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/05/17 14:27:20 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/17 14:27:23 INFO input.FileInputFormat: Total input paths to process : 5
17/05/17 14:27:23 INFO input.FileInputFormat: Total input paths to process : 5
17/05/17 14:27:23 INFO mapreduce.JobSubmitter: number of splits:4
17/05/17 14:27:23 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
17/05/17 14:27:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1495052924480_0004
17/05/17 14:27:24 INFO impl.YarnClientImpl: Submitted application application_1495052924480_0004
17/05/17 14:27:24 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1495052924480_0004/
17/05/17 14:27:24 INFO mapreduce.Job: Running job: job_1495052924480_0004
17/05/17 14:27:34 INFO mapreduce.Job: Job job_1495052924480_0004 running in uber mode : false
17/05/17 14:27:34 INFO mapreduce.Job: map 0% reduce 0%
17/05/17 14:27:41 INFO mapreduce.Job: map 25% reduce 0%
17/05/17 14:27:42 INFO mapreduce.Job: map 100% reduce 0%
17/05/17 14:27:42 INFO mapreduce.Job: Job job_1495052924480_0004 failed with state FAILED due to: Task failed task_1495052924480_0004_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

17/05/17 14:27:42 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=1
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=4838
Total vcore-seconds taken by all map tasks=4838
Total megabyte-seconds taken by all map tasks=1238528
17/05/17 14:27:42 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
17/05/17 14:27:42 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 21.6501 seconds (0 bytes/sec)
17/05/17 14:27:42 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
17/05/17 14:27:42 INFO mapreduce.ExportJobBase: Exported 0 records.
17/05/17 14:27:42 ERROR tool.ExportTool: Error during export: Export job failed!
[training@localhost ~]$ sqoop export --connect jdbc:mysql://localhost/loudacre --username training --password training --table device_export --export-dir /user/hive/warehouse/sqoop_import.db/device
17/05/17 14:32:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
17/05/17 14:32:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/05/17 14:32:41 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/05/17 14:32:41 INFO tool.CodeGenTool: Beginning code generation
17/05/17 14:32:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM device_export AS t LIMIT 1
17/05/17 14:32:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM device_export AS t LIMIT 1
17/05/17 14:32:42 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-training/compile/aeb8cb1f56f3398bd60dccf53ef98c31/device_export.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/05/17 14:32:45 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-training/compile/aeb8cb1f56f3398bd60dccf53ef98c31/device_export.jar
17/05/17 14:32:45 INFO mapreduce.ExportJobBase: Beginning export of device_export
17/05/17 14:32:45 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
17/05/17 14:32:45 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/05/17 14:32:45 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts
17/05/17 14:32:47 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
17/05/17 14:32:47 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
17/05/17 14:32:47 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/05/17 14:32:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/17 14:32:50 INFO input.FileInputFormat: Total input paths to process : 4
17/05/17 14:32:50 INFO input.FileInputFormat: Total input paths to process : 4
17/05/17 14:32:50 INFO mapreduce.JobSubmitter: number of splits:3
17/05/17 14:32:50 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
17/05/17 14:32:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1495052924480_0005
17/05/17 14:32:51 INFO impl.YarnClientImpl: Submitted application application_1495052924480_0005
17/05/17 14:32:51 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1495052924480_0005/
17/05/17 14:32:51 INFO mapreduce.Job: Running job: job_1495052924480_0005
17/05/17 14:33:01 INFO mapreduce.Job: Job job_1495052924480_0005 running in uber mode : false
17/05/17 14:33:01 INFO mapreduce.Job: map 0% reduce 0%
17/05/17 14:33:09 INFO mapreduce.Job: map 100% reduce 0%
17/05/17 14:33:09 INFO mapreduce.Job: Job job_1495052924480_0005 failed with state FAILED due to: Task failed task_1495052924480_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

17/05/17 14:33:09 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=1
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=4530
Total vcore-seconds taken by all map tasks=4530
Total megabyte-seconds taken by all map tasks=1159680
17/05/17 14:33:09 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
17/05/17 14:33:09 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 21.7892 seconds (0 bytes/sec)
17/05/17 14:33:09 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
17/05/17 14:33:09 INFO mapreduce.ExportJobBase: Exported 0 records.
17/05/17 14:33:09 ERROR tool.ExportTool: Error during export: Export job failed!


#2

Please go through above link to get the details of error.


#3

I was able to resolve this by using the correct delimiter into query:
–input-fields-terminated-by ‘\001’


#4

great so the problem was on your input file to load it to MySQL using Sqoop cause each field was delimited by ‘\001’ i would like to see your sample file . if you can upload a screenshot is going to be awesome lol thanks


#5

Hello @Andres_Angel, I’ve had the same issue but the issue can be because of multiple reasons, mine was order_date was too long in Hiveand the table field in mysql was only varchar(20), so it was failing, I increased the size to 100 (varchar(100)) and it worked perfectly fine.