Ideas wanted for analyzing near-realtime data over specific intervals with memory/cpu efficiency

Question

I have some environmental sensors and I want to detect sudden changes in temperature, and slow trends over time... however I'd like to do most of the math based on what's in memory with parameters that may look like this: (subject to change)

(note: Items in parens are computed in realtime as data is added)

• 5 minute (derivative, max, min, avg) + 36 datapoints for most current 3 hours
• 10 minute (derivative, max, min, avg) + 0 datapoints, calc is based on 5min sample
• 30 minute (derivative, max, min, avg) + 0 datapoints, calc is based on 5 min sample
• Hourly (derivative, max, min, avg) + 24 datapoints for most current 1 day
• Daily (derivative, max,min,avg) + 32 datapoints for most current month
• Monthly (derivative, max,min,avg) + 12 datapoints for past year

Each datapoint is a two byte float. So each sensor will consume up to 124 Floats, plus the 24 calculated variables. I'd like to support as many sensors as the .NET embededd device will permit.

Since I'm using an embedded device for this project, my memory is constrained and so is my IO and CPU power.

How would you go about implementing this in .NET? So far, I've created a couple of structs and called it a "TrackableFloat" where the insertion of a value causes the old one to drop off the array and a recalculation is done.

The only thing that makes this more complicated than it would be, is that for any sensor does not report back data, then that datapoint needs to be excluded/ignored from all subsequent realtime calulations.

When all is said and done, if any of the values: (derivative, max,min,avg) reach a pre defined setting, then a .NET event fires

I think someone out there will think this is an interesting problem, and would love to hear how they would approach implementing it.

1. Would you use a Class or a Struct?

2. How would you trigger the calculations? (Events most likely)

3. How would the alerts be triggered?

4. How would you store the data, in tiers?

5. Is there a library that already does something like this? (maybe that should be my first question )

6. How would you efficiently calculate the derivative?

Here is my first crack at this, and it doesn't completely hit the spec, but is very efficient. Would be interested in hearing your thoughts.

enum UnitToTrackEnum
{
    Minute,
    FiveMinute,
    TenMinute,
    FifteenMinute,
    Hour,
    Day,
    Week,
    Month,
    unknown
}
class TrackableFloat
{
    object Gate = new object();

    UnitToTrackEnum trackingMode = UnitToTrackEnum.unknown;
    int ValidFloats = 0;
    float[] FloatsToTrack;

    public TrackableFloat(int HistoryLength, UnitToTrackEnum unitToTrack)
    {
        if (unitToTrack == UnitToTrackEnum.unknown)
            throw new InvalidOperationException("You must not have an unknown measure of time to track.");

        FloatsToTrack = new float[HistoryLength];

        foreach (var i in FloatsToTrack)
        {
            float[i] = float.MaxValue;
        }


        trackingMode = unitToTrack;

        Min = float.MaxValue;
        Max = float.MinValue;
        Sum = 0;
    }
    public void Add(DateTime dt, float value)
    {
        int RoundedDTUnit = 0;

        switch (trackingMode)
        {
            case UnitToTrackEnum.Minute:
                {
                    RoundedDTUnit = dt.Minute;
                    break;
                }
            case UnitToTrackEnum.FiveMinute:
                {
                    RoundedDTUnit = System.Math.Abs(dt.Minute / 5);
                    break;
                }
            case UnitToTrackEnum.TenMinute:
                {
                    RoundedDTUnit = System.Math.Abs(dt.Minute / 10);
                    break;
                }
            case UnitToTrackEnum.FifteenMinute:
                {
                    RoundedDTUnit = System.Math.Abs(dt.Minute / 15);
                    break;
                }
            case UnitToTrackEnum.Hour:
                {
                    RoundedDTUnit = dt.Hour;
                    break;
                }
            case UnitToTrackEnum.Day:
                {
                    RoundedDTUnit = dt.Day;
                    break;
                }
            case UnitToTrackEnum.Week:
                {
                    //RoundedDTUnit = System.Math.Abs( );
                    break;
                }
            case UnitToTrackEnum.Month:
                {
                    RoundedDTUnit = dt.Month;
                    break;
                }
            case UnitToTrackEnum.unknown:
                {
                    throw new InvalidOperationException("You must not have an unknown measure of time to track.");
                }
            default:
                break;
        }


        bool DoRefreshMaxMin = false;
        if (FloatsToTrack.Length < RoundedDTUnit)
        {
            if (value == float.MaxValue || value == float.MinValue)
            {
                // If invalid data...
                lock (Gate)
                {
                    // Get rid of old data...
                    var OldValue = FloatsToTrack[RoundedDTUnit];
                    if (OldValue != float.MaxValue || OldValue != float.MinValue)
                    {
                        Sum -= OldValue;
                        ValidFloats--;

                        if (OldValue == Max || OldValue == Min)
                            DoRefreshMaxMin = true;
                    }

                    // Save new data
                    FloatsToTrack[RoundedDTUnit] = value;
                }
            }
            else
            {
                lock (Gate)
                {
                    // Get rid of old data...
                    var OldValue = FloatsToTrack[RoundedDTUnit];
                    if (OldValue != float.MaxValue || OldValue != float.MinValue)
                    {
                        Sum -= OldValue;
                        ValidFloats--;
                    }

                    // Save new data
                    FloatsToTrack[RoundedDTUnit] = value;
                    Sum += value;
                    ValidFloats++;

                    if (value < Min)
                        Min = value;

                    if (value > Max)
                        Max = value;

                    if (OldValue == Max || OldValue == Min)
                        DoRefreshMaxMin = true;
                }
            }

            // Function is placed here to avoid a deadlock
            if (DoRefreshMaxMin == true)
                RefreshMaxMin();
        }
        else
        {
            throw new IndexOutOfRangeException("Index " + RoundedDTUnit + " is out of range for tracking mode: " + trackingMode.ToString());
        }
    }

    public float Sum { get; set; }
    public float Average
    {
        get
        {
            if (ValidFloats > 0)
                return Sum / ValidFloats;
            else
                return float.MaxValue;
        }
    }
    public float Min { get; set; }
    public float Max { get; set; }
    public float Derivative { get; set; }

    public void RefreshCounters()
    {
        lock (Gate)
        {
            float sum = 0;
            ValidFloats = 0;

            Min = float.MaxValue;
            Max = float.MinValue;

            foreach (var i in FloatsToTrack)
            {
                if (i != float.MaxValue || i != float.MinValue)
                {
                    if (Min == float.MaxValue)
                    {
                        Min = i;
                        Max = i;
                    }

                    sum += i;
                    ValidFloats++;

                    if (i < Min)
                        Min = i;

                    if (i > Max)
                        Max = i;
                }
            }
            Sum = sum;
        }
    }
    public void RefreshMaxMin()
    {
        if (ValidFloats > 0)
        {
            Min = float.MaxValue;
            Max = float.MinValue;

            lock (Gate)
            {
                foreach (var i in FloatsToTrack)
                {
                    if (i != float.MaxValue || i != float.MinValue)
                    {
                        if (i < Min)
                            Min = i;

                        if (i > Max)
                            Max = i;
                    }
                }
            }
        }
    }
}

Answer 1

You should consider looking at a CEP library like Nesper.