Is it possible to view logs during certification?

Is it possible to view logs during certification?
4.7 3

#1

If we just invoke the following spark shell during the certification exam, where and how do we view the logs?
pyspark --master yarn


#2

Which Logs you want? you mean the resource manager which has cores and executors etc.?


#3

Whenever there is a failed job or an exception


#4

Yes. You can. I have used mapred job to debug sqoop export failures.


#5

Did you use the tracking URL to see the reason for failure ? I have heard that you cannot access that URL. Can you please elaborate on how you checked the sqoop export logs?


#6

I did not use the URL. I have used mapred job command to debug the sqoop export failures.


#7

Thanks. I tried the following command.
mapred job -logs job_1525279861629_9203

@dgadiraju - Can you share any troubleshooting tips and tricks in terms of checking logs, common mistakes etc?


#8

I have tried and it is working fine in lab.

mapred job -logs job_id task_id
ex., mapred job -logs job_1513875462555_0018 attempt_1513875462555_0018_m_000000_0

mapred job -logs job_1500459105861_9355 attempt_1500459105861_9355_m_000003_0

–You will see the details like below…

java.lang.RuntimeException: Can’t parse input data: ‘430542014-04-18 00:00:00.04914CLOSED’
at orders_sara.__loadFromFields(orders_sara.java:365)
at orders_sara.parse(orders_sara.java:298)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.NumberFormatException: For input string: “430542014-04-18 00:00:00.04914CLOSED”