How to write avro for multiple output directories using spark

Question

Hi, There is a topic about writing text data in multiple output directories in one spark job using MultipleTextOutputFormat

Recording on multiple outputs using a Spark key - one Spark job

I would ask if there is a similar way to write avro data to multiple directories

I want to write the data in the avro file to another directory (according to the timestamp field, it is sent to the same directory on the same day in timestamp)

Answer 1

The AvroMultipleOutputs class makes it easy to write Avro output to multiple outputs.

• Case one: write to additional outputs, except for the default output. Each additional output or named output can be configured using its own circuit and output formats.

• Case two: to write data to different files provided by the user

AvroMultipleOutputs supports counters; they are disabled by default. The counter group is the name of the AvroMultipleOutputs class. The names of the counters match the names of the outputs. They count the number of records recorded for each output name.

Also look

• MultipleOutputer
• MultipleOutputsFormatTest (see sample code with unit test argument here ... For some reason, MultipleOutputs does not work with Avro, but almost identical AvroMultipleOutputs. These explicitly related classes do not have a common ancestor, so they are combined in a class like MultipleOutputer, which, by at least allows further expansion. )