I am working on what is basically a highly accessible distributed messaging system. The system receives messages from any place via HTTP or TCP, performs various conversions on it, and then sends them to one or more recipients (also using TCP / HTTP).
There is a requirement in the system that all messages sent to the given address should be in order, because some messages were based on the contents of the previous ones. This limits us to processing messages sequentially, which takes about 750 ms per message. Therefore, if someone sends us, for example, one message every 250 ms, we are forced to queue messages one after another. This ultimately leads to an unbearable delay in processing messages under high load, as each message may have to wait until hundreds of other messages are processed before it arrives.
To solve this problem, I want to be able to parallelize our message processing without violating the requirements that we send them in order.
We can easily scale our processing horizontally. The missing part is a way to ensure that even if messages are processed out of order, they are “reordered” and sent to recipients in the order in which they were received. I am trying to find the best way to achieve this.
Apache Camel thing called Resequencer , which does this, and includes a beautiful diagram (which I don’t have enough to insert directly). This is exactly what I want: something that displays messages out of order and puts them in order.
But I do not want this to be written in Java, and I need the solution to be very affordable (i.e. resistant to typical system crashes such as system crashes or reboots), which I don't think Apache Camel offers.
Node.js, Redis Postgresql . Kue . Kue , , , , Kue .
, , . , , . , " ", " ", " " .., , " " - . . , :
- .
- , , , .
- Node.js
- HA (.. , .)
, , , Redis , . , :
Node, API -queue-esque, Redis. , , : ( , Node/Redis stack, ), out- ? - , ? !