All geek questions in one place

Clearre mapreduce task with avro output

I am rather confused that I use Avro with map reduction and cannot find suitable tutorials.

Classes like AvroJob and AvroMapper seem to be designed to solve problems when input and output are Avro data files. How about when your input is plain text?

In particular:

My cartographer accepts LongWritable and Text keys as input. It emits text keys and MyAvroRecord values.

My reducer accepts text keys and Iterator MyAvroRecords as input and emits text keys and MyAvroRecord values.

How to get an OutputFormat that will write these text keys and MyAvroRecord values ​​to a file?

Cheers, dave

+6
source share
2 answers

Ok, so I figured it out.

Instead of a display that displays text keys and MyAvroRecord values, I need one that generated AvroKey keys and AvroValue values. This managed to pass the results directly to AvroReducer, and I could just use AvroJob.setOutputSchema () to process the output (I didn't need to implement OutputFormat at all).

+6
source

Another approach may be: the output of the display device should not be AvroKey and AvroValue. These may be your common output types that contribute to your gearbox. In the gearbox, we can do the Avro conversion. By setting the type of Outputformat to Avro.

Regards, sujoy

0
source

Source: https://habr.com/ru/post/910765/

More articles:


All Articles