There is an API for JobTracker , as described here , which gives you a ton of information about the cluster itself, as well as details for all jobs.
In particular, if you know the task identifier and want to find indicators for each individual map and shorten tasks, you can call getMapTaskReports , which will return an instance of TaskReport , detailed here , which gives you access to methods such as getFinishTime or getStartTime . For example:
TaskReport[] maps = jobtracker.getMapTaskReports("your_job_id"); for (TaskReport rpt : maps) { long duration = rpt.getFinishTime() - rpt.getStartTime(); System.out.println("Mapper duration: " + duration); } TaskReport[] reduces = jobtracker.getReduceTaskReports("your_job_id"); for (TaskReport rpt : reduces) { long duration = rpt.getFinishTime() - rpt.getStartTime(); System.out.println("Reducer duration: " + duration); }
To calculate the total time for all the mappers or gearboxes in your work, you can simply sum them up simply in code.
And with regard to shuffling, this is usually taken into account by jobtracker as 33% of each reduction task, which does not necessarily mean it is 33% of the time, but I don’t think there is an automatic way to get shuffling time so that you can just go with this simple heuristic with 33% .
Please note that using time measurements from the jobtracker API, as shown above, the time in the gearboxes can be a little biased, because when the reduction task starts, there is essentially a shuffle (up to 33%, as explained), then it expects until all tasks of the card are completed, and only then it will begin the actual reduction, so the decrease in measurement is actually the sum of these three periods (shuffle + wait + decrease).