.avsc file not found after sqoop import using --as-avrodatafile

#1

After successful sqoop import using --as-avrodatafile, the .avsc file is not found under the home directory /home/cloudera, though the avro files got created in the target-dir. Appreciate the help! - vsu

0 Likes

#2

avsc file will be there in the directory from where you executed sqoop import command. Not necessarily in your home directory unless you ran sqoop import from there.

0 Likes

#3

Thanks! In fact I executed it from the home directory but still its not there. On checking the import result log, saw the .avsc file got created in a temporary directory “/tmp/sqoop-cloudera/compile/fe18e47d969e12365b846c4e20ffe03/”. I used the same for creating hive table.

1 Like

#4

Interesting, can you paste your complete sqoop import code?

0 Likes

#5

Here’s the import code:

sqoop import
–connect “jdbc:mysql://quickstart.cloudera:3306/retail_db”
–username retail_dba
–password cloudera
–table departments
–target-dir /user/cloudera/departments
–as-avrodatafile
–outdir java_files

0 Likes

#6

Which version of cloudera VM you are using? Also please review configuration files under /etc/sqoop/conf, probably there could be some parameter which is controlling about where files should be stored.

0 Likes

#7

Using cloudera-quickstart-vm-5.8.0-0-vmware. Under /etc/sqoop/conf, the following is found:
drwxr-xr-x 2 root root 4096 Dec 11 08:33 managers.d
-rw-r–r-- 1 root root 288 Dec 11 08:33 sqoop-site.xml
-rw-r–r-- 1 root root 26 Dec 11 08:33 cloudera_metadata
-rw-r–r-- 1 root root 20 Dec 11 08:33 cloudera_generation
-rw-r–r-- 1 root root 139 Dec 11 08:33 sqoop-env.sh

Of which sqoop-env.sh has:
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH::

0 Likes

#8

Check both .sh as well as .xml files to see if any parameter have value starting with /tmp

0 Likes

#9

Content of sqoop-env.sh:
"!/usr/bin/env bash
Generated by Cloudera Manager and should not be modified directly
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:: "

Content of sqoop-site.xml:
Just has the name tags - “sqoop.connection.factories” and “sqoop.tool.plugins”

Don’t find any parameter having value starting with /tmp.

0 Likes

#10

Hmm, not sure. It is tough to troubleshoot the issue as I do not have Cloudera Quickstart VM.

0 Likes

Avro data file extension is not avsc
#11

No Problem… Thanks Sir!

0 Likes