Hadoop: how to learn how to reduce partition_Id in a step using the Context object

Question

In the Hadoop ver. 0.20 and above, a Context object was introduced instead of JobConf.

I need to find the Context object:

• partition_id for current gear

• output folder

Using the deprecated JobConf, I can find the partition_id for the current gearbox as follows:

public void configure(JobConf conf){
  int  current_partition = conf.getInt("mapred.task.partition",-1);
}

I think that using the Context object, I need to do this inside the method

public void setup(Context c)

But how? What about the name of the output folder?

Answer 1

You can try to simply execute the Partitioner class on the first key of the input data, and in the end you will get several current reducers.

" " . , OutputFormat, , " " , , RDBMS - SQL. HDFS , JobContext, ..

c.getConfiguration().get("mapred.output.dir")

, URL .

Answer 2

, context.getTaskAttemptID(). getTaskID(). getId(). . , ReduceTaskImpl, TaskImpl MRBuilderUtils .

public TaskImpl(JobId jobId, TaskType taskType, int partition,
    EventHandler eventHandler, Path remoteJobConfFile, JobConf conf,
    TaskAttemptListener taskAttemptListener, OutputCommitter committer,
    Token jobToken,
    Credentials credentials, Clock clock,
    Map completedTasksFromPreviousRun, int startCount,
    MRAppMetrics metrics, AppContext appContext) {
  this.conf = conf;
  this.clock = clock;
  this.jobFile = remoteJobConfFile;
  ReadWriteLock readWriteLock = new ReentrantReadWriteLock();
  readLock = readWriteLock.readLock();
  writeLock = readWriteLock.writeLock();
  this.attempts = Collections.emptyMap();
  maxAttempts = getMaxAttempts();
  taskId = MRBuilderUtils.newTaskId(jobId, partition, taskType);
  this.partition = partition;
  ...
}

public static TaskId newTaskId(JobId jobId, int id, TaskType taskType) {
  TaskId taskId = Records.newRecord(TaskId.class);
  taskId.setJobId(jobId);
  taskId.setId(id);
  taskId.setTaskType(taskType);
  return taskId;
}