Combine without gear in Hadoop

Question

Can I write Hadoop code that only has Mappers and Combiners (i.e. mini gearboxes without gearbox)?

job.setMapperClass (WordCountMapper.class);
job.setCombinerClass (WordCountReducer.class);

conf.setInt ("mapred.reduce.tasks", 0);

I tried to do this, but I always see that I have one shortcut task on the job tracking link

Running Reduce Tasks = 1

How can I remove reducers while maintaining combinators? perhaps?

Answer 1

In this case, you should use Reducers. Use as key: Context.getInputSplit (). GetPath () + Context.getInputSplit (). GetStart () - This combination is unique to each Mapper.

Answer 2

You need to tell your work that you don't care about the gearbox: JobConf.html # setNumReduceTasks (int)

// new Hadoop API jobConf.setNumReduceTasks(0); // old Hadoop API job.setNumReduceTasks(0); 

You can accomplish something with IdentityReducer .

Performs a reduction by writing all input values ​​directly to the output.

I'm not sure you can continue to combine, but I will start from the previous lines.