Hierarchical rule triggering with Esper

Question

We have a stream of events each having the following properties:

public class Event {
    private String id;
    private String src;
    private String dst;
}

Besides, we have a set of hierarchical or nested rules we want to model with EPL and Esper. Each rule should be applied if and only if all of its parent rules have been already activated (a matching instance occurred for all of them). For example:

2 events or more with the same src and dst in 10 seconds
  + 5 or more with src, dst the same as the src, dst in the above rule in 20s
    + 100 or more with src, dst the same as the src, dst in the above rules in 30s

We want to retrieve all event instances corresponding to each level of this rule hierarchy. For example, considering following events:

id ---- source -------------- destination ---------------- arrival time (second)
1     192.168.1.1             192.168.1.2                      1
2     192.168.1.1             192.168.1.2                      2
3     192.168.1.1             192.168.1.3                      3
4     192.168.1.1             192.168.1.2                      4
5     192.168.1.5             192.168.1.8                      5
6     192.168.1.1             192.168.1.2                      6
7     192.168.1.1             192.168.1.2                      7
8     192.168.1.1             192.168.1.2                      8
.....
100 other events from 192.168.1.1 to 192.168.1.2 in less than 20 seconds

We want our rule hierarchy to report this instance together with the id of all events corresponding to each level of the hierarchy. For example, something like the following report is required:

2 or more events with src 1928.168.1.1 and dst 192.168.1.2 in 10 seconds ( Ids:1,2 )
  + 5 or more with the same src (192.168.1.1) and dst (192.168.1.2) in 20s (Ids:1,2,4,6,7)
        + 100 or more events from 192.168.1.1 to 192.168.1.2 in 30s (Ids:1,2,4,6,7,8,...)

How can we achieve this (retrieve the ids of the events matched with all rules) in Esper EPL?

Answer 1

Complex use case, it will take some time to model this. I'd start simple with keeping a named window and using some match-recognize or EPL patterns. For the rule nesting, I'd propose triggering other statements using insert-into. A context partition that gets initiated by a triggering event may also come in handy. For the events that are shared between rules, if any, go against the named window using a join or subquery, for example. For the events that arrive after a triggering event of the first or second rule, just use EPL statements that consume the triggering event. Start simple and build it up, become familiar with insert-into and declaring an overlapping context.

Answer 2

You could use each rule as the input for the next rule in the hierarchy, for example, the rule listens for matching events in the last 10 secs ans inserts the results with the 'insert into' clause to a new stream, so the next rule is triggered for events in the new steam and so on... it it a pretty simple use case and can be done even without context partitions.