Processing header files in Hadoop

Question

I want to process many files in Hadoop - each file has some header information, followed by many records, each of which is stored in a fixed number of bytes. Any suggestions on this?

Answer 1

I think the best solution is to create a custom one InputFormat.

Answer 2

There is one solution, you can check the offset of the lines of the files that the cartographer reads. It will be zero for the first line in the file. therefore, you can add a line on the map as follows:

public void map ( LongWritable, , ) throws IOException, InterruptedException       {

        if(key.get() > 0)
                       {
                         your mapper code
                       }
              }

, .

, , .

-

Answer 3

In addition to writing a custom FileInputFormat, you will also want to make sure that the file is not corrupted, so the reader knows how to handle the entries inside the file.