Map time value or time reduction in JobHistoryServer

I want to know the exact meaning of the notation in the figure below. This image came from the job history serverweb interface. I definitely know the meaning of the Expired, but I'm not sure about other things. Where can I find a clear definition? Or is there anyone who knows the meaning of these?

What I want to know is card time, reduce time, shuffle time and merge time separately. And the sum of the four times should be very similar (or equal) to the elapsed time. But the keyword "Average" makes me embarrassed.

Screenshot from Job history server

There are 396 cards and 1 decrease.

+4
source share
1 answer

As you probably already know, there are three steps to setting MapReduce:

  • - 1- , , . "" , .

  • Shuffle - , , Map, . , . Shuffle , , . ( ) , . , Map → Reduce tasks.

  • - MapReduce. "" , , (HDFS/Hive/Hbase).

, , , 396 . , . , - , 396 .

Average Map Time = Total time taken by all Map tasks/ Number of Map Tasks

,

Average Reduce Time = Total time taken by all Reduce tasks/Number of Reduce tasks

, ? , , , ( / node ..). , "" "" .

, Shuffle 40 . .

  • 396 , . , , , . , , .

  • . , , , .

, , , , / , , , Heartbeat JobTracker, , NameNode, .., .

, .

+5

Source: https://habr.com/ru/post/1535682/


All Articles