Sqoop job fails at oozie workflow

sqoop
oozie
mysql

#1

Hi, I’ve created Workflow management at Oozie view on Hortonworks distribution. For testing, I’m trying to import mysql data thru sqoop to hdfs. This is the error I’m getting:
Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
sqoop command:
import --connect jdbc:mysql://nn01.itversity.com:3306/retail_export --username retail_dba --password itversity --table customerdata --target-dir /user/saikiranchepuri35/customerdata5 --m 4
Following files are being used for workflow.

job.properties
nameNode=hdfs://nn01.itversity.com:8020
jobTracker:8050=rm01.itversity.com
queueName=default
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}/customerdata5

workflow.xml




${resourceManager}
${nameNode}
import --connect jdbc:mysql://nn01.itversity.com:3306/retail_export --username retail_dba --password itversity --table customerdata --target-dir /user/saikiranchepuri35/customerdata5 --m 1





${wf:errorMessage(wf:lastErrorNode())}


Plz help me on this. Is there any issue giving job.properties on namenode and jobtracker port numbers?


#2

I dont think retail_dba user is accessible to all

Try with retail_user


#3

The variable which you have given is not defined in job.properties


#4

I’ve tried with “retail_user” also, but not working at all…


#5

This way I’ve given its port number for ${resourceManager} during “validate workflow” stage. But it is not working at all…


#6

@saikiranchepuri35 Can you try below simple sqoop with Ozzie example.

step1 : Create job.properties file as below:

nameNode=hdfs://nn01.itversity.com:8020
jobTracker=rm01.itversity.com:8050
queueName=default
oozie.use.system.libpath=true
oozie.libpath=${nameNode}/user/oozie/share/lib/lib_20180803155221/sqoop/
oozie.wf.application.path=${nameNode}/user/${user.name}/oozie_demo/simple_sqoop_demo

step2 : Create workflow.xml file as below:

<action name="sqoop-node">
    <sqoop xmlns="uri:oozie:sqoop-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <prepare>
            <delete path="${nameNode}/user/${wf:user()}/sqoop_script/"/>
        </prepare>
        <configuration>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
        </configuration>
        <command>import --connect "jdbc:mysql://ms.itversity.com/retail_db" --username retail_user --password itversity --table orders --driver com.mysql.jdbc.Driver --warehouse-dir /user/username/sqoop_script/ -m 1</command>
    </sqoop>
    <ok to="end"/>
    <error to="fail"/>
</action>

<kill name="fail">
    <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>

step3: Create a directory and copy a workflow.xml, job.properties files in your HDFS path:

hdfs dfs -mkdir /user/username/oozie_demo/simple_sqoop_demo
hdfs dfs -copyFromLocal job.properties workflow.xml /user/username/oozie_demo/simple_sqoop_demo

step4 : Run oozie job.properties:
oozie job -oozie http://nn01.itversity.com:11000/oozie -config /home/username/oozie_demo/pig_demo/job.properties

step5 : verify output from hdfs path:

hdfs dfs -ls /user/username/sqoop_script/


#8

Hi @annapurna. Thanks for your inputs. After executing the code snippet in above one, my job is getting into “SUSPENDED” state and below is the error i was getting.

2018-12-01 07:09:35,558 WARN ActionStartXCommand:523 - SERVER[nn01.itversity.com] USER[saikiranchepuri35] GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000050-181025050438919-oozie-oozi-W] ACTION[0000050-181025050438919-oozie-oozi-W@sqoop-node] Error starting action [sqoop-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.]
org.apache.oozie.action.ActionExecutorException: JA009: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:457)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:437)

Could you please update on it quickly. I couldn’t understand why it is throwing out like this.


#9

Any update on this?:confused:


#10

@saikiranchepuri35 Apologies for a last reply.

I have changed job.properties as below. Try now and lets us know.

nameNode=hdfs://nn01.itversity.com:8020
jobTracker=rm01.itversity.com:8050
queueName=default
oozie.use.system.libpath=true
oozie.libpath=${nameNode}/user/oozie/share/lib/lib_20180803155221/sqoop/
oozie.wf.application.path=${nameNode}/user/${user.name}/oozie_demo/simple_sqoop_demo

#11

It looks same as the earlier post. same job.properties been posted.