I’m unable to run the Hive query and it stuck at Mapreduce phase stage -2 Mapper 100% and Reducer 33%.
Earlier,I was also getting a GC overlimit Exhaust , Out of memory error also . I have set below parameter to hive query and still stuck at Mapper and reduce 33 %
set mapreduce.child.java.opts="-XX:NewRatio=12 -Xmx4096m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseGCOverheadLimit -XX:+UseConcMarkSweepGC"
Reducer is set to 1 by default.
Jobtracker Logs snip:
INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 100 segments left of total size: 141733920768 bytes . Stuck at this log point
Note: Data size may be huge in insert query.
Do I need to set sort.mb parameter in Hive or Set the number of reducer in hive or increase reducer size from 256 to 1G or Any alternative fix for this please . Kindly provide the exact numbers to set
Trying for more than two months to get the solution .
NN RAM : 4T
DN Ram : 90G *40