Is CEP what I need (system state and event replay)

Question

I am looking for a CEP mechanism, but I do not know if any engine meets my requirements. My system needs to process several streams of event data and generate complex events, and this is exactly what is ideal for any CEP mechanism (ESPER, Drools).

I store all unprocessed events in a database (this is not part of CEP, but I do) and use rules (or continuous requests or something else) to create custom actions for complex events. But some of my rules depend on past events. For example: I could send a sensor every time my spouse arrives or leaves home, and if my car and my fashionable woman’s car are near my house, I receive a “Dangerous” SMS.

The problem is that when I restart the event processing service, I lose all information about the state of the system (is my wife at home?), And to restore it I need to replay events for an unknown period of time. The state of the system may depend not only on raw events, but also on complex events.

The same problem occurs when I need a report on complex events in the past. I have raw event data stored in a database and can generate these complex events that reproduce raw events, but I don’t know for what period I have to reproduce them.

At the same time, it is clear that for most rules, you can automatically find the number of events that must be processed from the past (or the time period for loading events to be processed) to restore the state of the system. If this action depends on my wife’s presence at home, the CEP should request the latest status change. If a report on complex events is requested, and a complex event depends on the average price for the previous period, all price change events for this period should be reproduced. And so on...

Should I miss something?

Answer 1

RuleCore CEP Server can solve your problems if I remember correctly. It does not lose state if you restart it, and it contains a virtual logic clock so that you can play back events using any concept of time.

Answer 2

I'm not sure if you have a question whether current CEP products are related to historical data with live events, but if you need it, Esper allows you to extract data from JDBC sources (which links your historical data to your live events) and reflect them in your EPL statements. I think you've already checked the Esper website, if not, you'll see that Esper has excellent documentation with lots of cookbook examples.

But even if you model your historical events after your live events, this does not solve your problem with choosing the right time frame, and, as you wrote, this time frame depends on the use.

Answer 3

As previous people noted, I don’t think your problem is really a problem with the engine, but rather from one use case. All engines that I am familiar with, including Drools Fusion and Esper, can join inbound events with historical data and / or status data requested upon request from an external source (such as a database). It seems to me that you will need to save the state (or “timestamp breakpoints”) when the corresponding change occurs, and reload the state on repeated launches, rather than repeat the events for an unknown time interval.

Alternatively, if you use Drools, you can check the existing rules (type of reflection of your rules / requests) to find out what types of events your rules need and cancel the event log until all requirements are met and upload / play your events there using session hours.

Finally, you can use the cluster to reduce restart, but this does not solve the problem you are describing.

Hope this helps.