Step 02 - Writing map reduce programs - Aggregations

This is 2nd 2 week plan for the preparation of HDPCD Java examination

Goal: Get comfortable in writing, executing and troubleshooting simple map reduce programs. Also make sure you understand HDFS commands

  • HDFS briefly
  • Introduction to map reduce APIs
  • Default Mappers and Reducers
  • Develop row count program
  • Run row count program on the cluster
  • Understand counters

Here is the playlist for the reference

Next Plan: Aggregations and role of combiners

I pulled the master branch from the git for mapreduce
I tried running it fails with the below

ps:Its windows machine

log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See for more info.
Exception in thread “main” java.lang.NullPointerException
at java.lang.ProcessBuilder.start(
at org.apache.hadoop.util.Shell.runCommand(
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
at org.apache.hadoop.util.Shell.execCommand(
at org.apache.hadoop.util.Shell.execCommand(
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
at org.apache.hadoop.mapreduce.Job$
at org.apache.hadoop.mapreduce.Job$
at Method)
at org.apache.hadoop.mapreduce.Job.submit(
at org.apache.hadoop.mapreduce.Job.waitForCompletion(
at nyse.RecordCount.main(

What are the parameters you have passed?

I ran the same program in my linux Cludera vm’s eclipse .My project is working. I passed the same parameters . Thanks

Do we have map side joins and reduce side joins video tutorial ? or any plans in the near future. I could only see distributed cache beacuse on the HDPCD-JAVA tasks does include map side join.

Yes, I will add videos soon.