haskell - Average floating point error using QuickCheck

Question

I am using QuickCheck-2.5.1.1 to do QA. I am testing two pure functions gold :: a -> Float and f :: a -> Float, where a instances Arbitrary.

Here gold is a reference calculation and f is a variation I am optimizing.

To date, most of my tests using quickcheck have been using tests like \a -> abs (gold a - f a) < 0.0001.

However, I would like to gather statistics along with checking the threshold, since knowing the average error and standard deviation are useful in guiding my design.

Is there any way to use QuickCheck to gather statistics like this?

---

Concrete example

To give a concrete example of the sort of thing I'm looking for, suppose I have the following two functions for approximating square roots:

-- Heron's method
heron :: Float -> Float
heron x = heron' 5 1
    where
      heron' n est
          | n > 0 = heron' (n-1) $ (est + (x/est)) / 2
          | otherwise = est

-- Fifth order Maclaurin series expansion
maclaurin :: Float -> Float
maclaurin x = 1 + (1/2) * (x - 1) - (1/8)*(x - 1)^2
                + (1/16)*(x - 1)^3 - (5/128)*(x - 1)^4
                + (7/256)*(x - 1)^5

A test for this might be:

test = quickCheck
       $ forAll (choose (1,2))
       $ \x -> abs (heron x - maclaurin x) < 0.02

So what I'd like to know, as a side-effect of the test, is the statistics on abs (heron x - maclaurin x) (such as the mean and standard deviation).

Answer 1

Thanks to the comments from Chris Kuklewicz and Ingo, I came up with the following that collects the statistics I want in my example:

resultToWeightList :: Result -> [(Double,Int)]
resultToWeightList r = [ (read s, n) | (s,n) <- labels r]

weightListMuSigma :: [(Double,Int)] -> (Double,Double)
weightListMuSigma wlst = (mu,sigma)  
  where 
    (weightSum,weightSqrSum,entryCount) = foldl addEntry (0,0,0) wlst
    addEntry (s,s2,c) (v,w) = (s + (v * w'), s2 + (v**2 * w'), c + w)
      where w' = fromIntegral w
    entryCount' = fromIntegral entryCount
    mu = weightSum / entryCount'
    var = weightSqrSum / entryCount' - mu**2
    sigma = sqrt var

quietCheckResult :: Testable prop => prop -> IO Result
quietCheckResult p = quickCheckWithResult args p
  where args = stdArgs { chatty = False }

test :: IO ()
test = do { r <- quietCheckResult $ forAll (choose (1,2)) test'
          ; let wlst = resultToWeightList r
          ; let (mu,sigma) = weightListMuSigma wlst 
          ; putStrLn $ "Average: " ++ show mu
          ; putStrLn $ "Standard Deviation: " ++ show sigma
          }
   where
     test' x = collect err (err < 0.1)
       where err = abs $ heron x - maclaurin x