Smoothing experimental data with piecewise functions

Question

I have a data set of a single measurement vs. time (about 3000 points). I'd like to smooth the data by fitting a curve through it. The experiment is a multi-stage physical process so I am pretty sure a single polynomial won't fit the whole set.

Therefore I'm looking at a piecewise series of polynomials. I'd like to specify how many polynomials are used. This seems to me to a fairly straightforward thing and I was hoping that there would be some pre-built library to do it. I've seen org.apache.commons.math3.fitting.PolynomialFitter in Apache Commons Math but it seems to only work with a single polynomial.

Can anyone suggest the best way to do this? Java preferred but I could work in Python.

Answer 1

In finmath lib there is a class called curve which implements some interpolation schemes (linear, spline, akima, etc.). These curves can provide their points as parameters to a solver and you can then use a global optimization (like a Levenberg Marquardt optimizer) to minimize the distance of your data to the curve (defining some preferred norm).

This is actually done in the "Curve Calibration" which is an application from mathematical finance. If you have as many points (parameters) in the curve as data you will likely get a perfect fit. If you have fewer points than data you get the best fit in your norm.

The Levenberg Marquardt in finmath lib is multithreaded and very fast (> 200 points are fitted in << 1 sec).

See

• the Curve class at http://svn.finmath.net/finmath%20lib/trunk/src/main/java/net/finmath/marketdata/model/curves/Curve.java
• and the LM optimizer at http://svn.finmath.net/finmath%20lib/trunk/src/main/java/net/finmath/optimizer/LevenbergMarquardt.java

Disclaimer: I am the/a developer of that library.

Note: I also like commons-math, but for the curve fitting I don't use it (yet), since I need(ed) some fitting properties specific to my application (mathematical finance).

(Edit)

Here is small demo: (Note: This demo requires finmath-lib 1.2.13 or the current 1.2.12-SNAPSHOT available at mvn.finmath.net or github.com/finmath/finmath-lib (it is not compatible with 1.2.12)

package net.finmath.tests.marketdata.curves;

import java.text.DecimalFormat;
import java.text.NumberFormat;

import org.junit.Test;

import net.finmath.marketdata.model.curves.Curve;
import net.finmath.marketdata.model.curves.CurveInterface;
import net.finmath.optimizer.LevenbergMarquardt;
import net.finmath.optimizer.SolverException;

/**
 * A short demo on how to use {@link net.finmath.marketdata.model.curves.Curve}.
 * 
 * @author Christian Fries
 */
public class CurveTest {

    private static NumberFormat numberFormat = new DecimalFormat("0.0000");

    /**
     * Run a short demo on how to use {@link net.finmath.marketdata.model.curves.Curve}.
     * 
     * @param args Not used.
     * @throws SolverException Thrown if optimizer fails.
     * @throws CloneNotSupportedException Thrown if curve cannot be cloned for optimization.
     */
    public static void main(String[] args) throws SolverException, CloneNotSupportedException {
        (new CurveTest()).testCurveFitting();
    }

    /**
     * Tests fitting of curve to given data.
     * 
     * @throws SolverException Thrown if optimizer fails.
     * @throws CloneNotSupportedException Thrown if curve cannot be cloned for optimization.
     */
    @Test
    public void testCurveFitting() throws SolverException, CloneNotSupportedException {

        /*
         * Build a curve (initial guess for our fitting problem, defines the times).
         */
        Curve.CurveBuilder curveBuilder = new Curve.CurveBuilder();

        curveBuilder.setInterpolationMethod(Curve.InterpolationMethod.LINEAR);
        curveBuilder.setExtrapolationMethod(Curve.ExtrapolationMethod.LINEAR);
        curveBuilder.setInterpolationEntity(Curve.InterpolationEntity.VALUE);

        // Add some points - which will not be fitted
        curveBuilder.addPoint(-1.0 /* time */, 1.0 /* value */, false /* isParameter */);
        curveBuilder.addPoint( 0.0 /* time */, 1.0 /* value */, false /* isParameter */);

        // Add some points - which will be fitted
        curveBuilder.addPoint( 0.5  /* time */, 2.0 /* value */, true /* isParameter */);
        curveBuilder.addPoint( 0.75 /* time */, 2.0 /* value */, true /* isParameter */);
        curveBuilder.addPoint( 1.0 /* time */, 2.0 /* value */, true /* isParameter */);
        curveBuilder.addPoint( 2.2 /* time */, 2.0 /* value */, true /* isParameter */);
        curveBuilder.addPoint( 3.0 /* time */, 2.0 /* value */, true /* isParameter */);

        final Curve curve = curveBuilder.build();

        /*
         * Create data to which the curve should be fitted to
         */
        final double[] givenTimes   = { 0.0,  0.5, 0.75, 1.0, 1.5, 1.75, 2.5 };
        final double[] givenValues  = { 3.5, 12.3, 13.2, 7.5, 5.5, 2.9,  4.4 };

        /*
         * Find a best fitting curve.
         */

        // Define the objective function
        LevenbergMarquardt optimizer = new LevenbergMarquardt(
                curve.getParameter()    /* initial parameters */,
                givenValues             /* target values */,
                100,                    /* max iterations */
                Runtime.getRuntime().availableProcessors() /* max number of threads */  
                ) {

            @Override
            public void setValues(double[] parameters, double[] values) throws SolverException {

                CurveInterface curveGuess = null;
                try {
                    curveGuess = curve.getCloneForParameter(parameters);
                } catch (CloneNotSupportedException e) {
                    throw new SolverException(e);
                }

                for(int valueIndex=0; valueIndex<values.length; valueIndex++) {
                    values[valueIndex] = curveGuess.getValue(givenTimes[valueIndex]);
                }
            }
        };

        // Fit the curve (find best parameters)
        optimizer.run();

        CurveInterface fittedCurve = curve.getCloneForParameter(optimizer.getBestFitParameters());

        // Print out fitted curve
        for(double time = -2.0; time < 5.0; time += 0.1) {
            System.out.println(numberFormat.format(time) + "\t" + numberFormat.format(fittedCurve.getValue(time)));
        }

        // Check fitted curve
        double errorSum = 0.0;
        for(int pointIndex = 0; pointIndex<givenTimes.length; pointIndex++) {
            errorSum += fittedCurve.getValue(givenTimes[pointIndex]) - givenValues[pointIndex];
        }
        System.out.println("Mean deviation: " + errorSum);

        /*
         * Test: With the given data, the fit cannot over come that at 0.0 we have an error of -2.5.
         * Hence we test if the mean deviation is -2.5 (the optimizer reduces the variance)
         */
        org.junit.Assert.assertTrue(Math.abs(errorSum - -2.5) < 1E-5);
    }
}

Answer 2

If you're looking for local regression, Commons Math implements it as LoessInterpolator. You'll get the end result as a "spline," a smooth sequence of piecewise cubic polynomials.