Max Hadoop Line Length

I am working on a Hadoop streaming thread for Amazon Elastic Map Reduce, and it includes serializing some binary objects and streaming them to Hadoop. Does Hadoop have a maximum line length for streaming input?

I started just testing with large and large lines, but decided that I would ask about it first.

+3
source share
1 answer

There seems to be no limit to the length of the string. Since I asked a question, I write code that serializes binary objects, encodes them in base64, and then puts them in a stream for processing. As a result, some of the lines are quite long. Headop chewed directly without complaint.

+4
source

Source: https://habr.com/ru/post/1751745/


All Articles