I am working on the conversion of an existing time series database system to a MapReduce model using Hadoop. The database system has both historical and real processing capabilities. So far, I have managed to translate batch processing functionality into Hadoop.
Unfortunately, when it comes to real-time processing, I see that there are some conceptual inconsistencies with the MapReduce model.
I can write my own implementation of the Hadoop InputFormat interface, which will continuously file card files with new data so that mappers can process and emit data continuously. However, since the reduce () method is not called until all the mappers have completed execution, my calculations will certainly get stuck at the matching stage.
I saw several messages that were mentioned mapred.reduce.slowstart.completed.maps, but, as I understand it, this only controls when the reducers start to extract data to their local destinations (shuffling) - the actual reduction method is called only after all the cartographers have completed their execution.
Of course, it is possible to imitate continuous execution, processing small batches of data at small time intervals, using a continuous stream of separate MR tasks, but this will lead to additional delays, which is unacceptable in my case.
I also considered using Storm or S4 , but before moving on, I need to make sure that this is beyond the scope of Hadoop.
In general, it seems that people were able to develop real-time Hadoop applications (such as Impala) or real-time processing solutions built on top of Hadoop. The question is how?