How to show fluctuating data smoothly and accurately?

Question

I am working on a project in which I have to read data from liquid level sensor installed in tank which measures fluid level and sends output in millivolt. The device has predefined ranges of voltage to decide fluid level from voltage. The tank might be fitted on moving vehicle as well. I have to represent this data on graph.

Now, the problem is that the data sent by some devices is very fluctuating as you can see in below screenshot. In order to show smooth line, I used linear regression by creating bunch of 30 minute duration data and applied it on each bunch. The result is as below. As you can see, linear regression works pretty well and gives me smooth line but, the problem is that the refill event is delayed by hours.

So, my question is what are the possible methods which I can use to by which I can show smooth line and refill events on actual time as well?

Answer 1

As @amon already noted, domain knowledge may help to find an appropriate function.

For a vehicle tank you need to consider two types of value change: slow fuel consumption and fast fill. There are also at least two types of measurement error sources: quick fluctuations due to movement (noise-like) and medium-length aberrations due to the vehicle being parked at an angle. You see both of these effects in your data. Temperature may also have an effect but to compensate for it would require measuring and calculating the contribution, which is probably overkill.

A relative simple algorithm would be to compute a low pass filter combined with an override when the change is over a threshold (which depends on the expected magnitude of fluctuations.)

Something like this at each step:

filtered_value = (measured_value > filtered_value + THRESHOLD)
    ? measured_value
    : (1 - FACTOR) * filtered_value + FACTOR * measured_value

where FACTOR is the low-pass smoothing factor, must be between 0 and 1, smaller values yield stronger smoothing, and THRESHOLD is the tank-filling threshold. The values for both need to be determined experimentally, there's no single "right" choice.

Note that this does not handle parking angle measurement error yet, but that is probably not significant enough to be a problem.

Answer 2

Exponential smoothing (Hans-Martin above) is a good way to go.

SmoothedValue = ((1 - α) * currentReading) + (α * PreviousSmoothedValue)

where α is the smoothing coefficient. Any spike (change in regular pattern, maybe caused by liquid movement at the time of measurement) will impact the series, but its values will be smoothed out over the next few ( depending on α ) values. To derive a suitable coefficient, you need to determine the sampling rate of your tank sensor. From your graph it appears to be more frequent than once every 10 seconds, maybe once every 2 seconds. Next choose a length of time for the smoothing. You suggest that the 30 minutes you used is unacceptably long, maybe 30 seconds would be acceptable as long as there is AT LEAST 5 measurements in the interval. After this time length any spike will be completely smoothed out. Then set (1 - α) to be the reciprocal of the number of measurements in this time interval.

The tank re-fill at 12:30 PM appears to happen quite fast within one measurement interval. If you are prepared to accept a narrow "S" shaped curve for this fill (instead of a vertical step) then you won't need the threshold test.

To start the process at time 0, just set the previousSmoothedValue to be the currentReading.