Hasoop + one key to each gearbox

Question

Is there a way in Hadoop to ensure that each gearbox receives only one key that is displayed by the display device?

Answer 1

This question is a bit unclear to me. But I think I have a good idea what you want.

First of all, if you do not do anything special with each reduction call, it receives only one single key with a set of one or more values ​​(via an iterator).

I assume that you want each gearbox to receive exactly one key-value pair. There are two ways to do this:

• Make sure all keys displayed are unique. Therefore, for each key there is only one value.
• , , .

, . GroupComparator, , .

---

- :

3 ( , API 0.18.3):

    conf.setPartitionerClass(KeyPartitioner.class);

, ", , ". 1 , , .

    conf.setOutputKeyComparatorClass(KeyComparator.class);

"-" , ... - .

    conf.setOutputValueGroupingComparator(GroupComparator.class);

, , .

Answer 2

, , Partitioner

API Hadoop:

Partitioner -. ( ) , . . , m (, , ) .

, : http://www.umiacs.umd.edu/~jimmylin/book.html

Answer 3

, , , , , . youtube mapreduce ucb 61a lecture-34, .

Answer 4

? , .

,

• , .
• , ,