Streaminsight maximum window length?

Question

I'm considering Streaminsight for use identifying patterns in driver behavior data that will include patterns that occur over both short (a few seconds) and long periods (over 1 week). I've struggled to find any information about the maximum window length or the resources required vs. window length or examples of using Complex event processing over longer periods.

The only indication of max window length I've found is that its a function of the stream data rate, payload size, memory and CPU but nothing exact. How can I calculate a maximum window length given the spec of the server and the amount of data it has to process?

Am I looking at the wrong technology for processing long temporal streams or is there a workaround involving manipulating the stream that would keep the resource requirements down?

Answer 1

I don't know any hard and fast rules on maximum window size. Are you processing your driver events in real-time or are you replaying driver events? If you are replaying, then you should be able to test that out easy enough through trial and error. You can use StreamInsight's performance counters to get some insight into how much memory is being consumed. You can load up StreamInsight quite a bit depending on the edition you are using. Premium will give you more schedulers to work with.

As far as tips go to reduce memory usage, I would tell you to filter out events that you do not need as soon as you can in your queries. If you don't need an event, there is no sense in letting it hang around. Also, in some cases, altering the event lifetime to something like DateTime.MaxLength can be a memory hog depending on how many events are coming through.