How to display a reason of a failed test property with quickcheck?

Question

What is the best practice to display reasons for a failed property test when it is tested via QuickCheck?

Consider for example:

prop a b = res /= []
   where
      (res, reason) = checkCode a b

Then the a session could look like:

> quickCheck prop
Falsifiable, after 48 tests:
42
23

But for debugging it would be really convenient to show the reason for failure as part of the quickCheck falsifable report.

I have hacked it like this:

prop a b = if res /= [] then traceShow reason False else True
   where
      (res, reason) = checkCode a b

Is there is a better/nicer or more quickcheckish way to do it?

Answer 1

I assume that your "reason" variable contains some kind of test-specific data on what went wrong. You could instead return a "Result", which contains both success/fail/invalid conditions and a string explaining what went wrong. Properties that return Results are handled by QuickCheck in exactly the same way as properties that return Bool.

(edit) Like this:

module QtTest where 

import Test.QuickCheck
import Test.QuickCheck.Property as P


-- Always return success
prop_one :: Integer -> P.Result
prop_one _ = MkResult (Just True) True "always succeeds" False [] []


-- Always return failure
prop_two :: Integer -> P.Result
prop_two n = MkResult (Just False) True ("always fails: n = " ++ show n) False [] []

Note that it is the "Result" type defined in Test.QuickCheck.Property you want.

There are also some combinators defined in Test.QuickCheck.Property which help you compose the Result rather than calling the constructor directly, such as

prop_three :: Integer -> Property
prop_three n = printTestCase ("always fails: n = " ++ show n) False

I guess it would be better style to use those.

Answer 2

Because QuickCheck gives you the inputs to the function, and because the code under test is pure (it is, right?), you can just feed those inputs to the function and get the result. This is more flexible, because with those inputs you can also repeatedly test with tweaks to the original function until it's correct.

Answer 3

This is my solution (I use counterexample instead of printTestCase since the later one is deprecated now):

(<?>) :: (Testable p) => p -> String -> Property
(<?>) = flip (Test.QuickCheck.counterexample . ("Extra Info: " ++))
infixl 2 <?>

Usage:

main :: IO ()
main = hspec $ do
  describe "math" $ do
    prop "sum-of-square-le-square-of-sum" $ do
      \(x :: Int) (y :: Int) ->
        x * x + y * y <= (x + y) * (x + y) <?> show (x * x, y * y, x + y)

So when a test case fails, you can see something like:

   *** Failed! Falsifiable, Falsifiable (after 2 tests):
   1
   -1
   Extra Info: (1,1,0)

You can also use <?> together with .&&., .||., === and ==> etc.:

  describe "math" $ do
    prop "sum-of-square-le-square-of-sum" $ do
      \(x :: Int) (y :: Int) ->
        x * x + y * y <= (x + y) * (x + y) <?> show (x * x, y * y, x + y) .||. (1==0) <?> "haha"

Answer 4

This works in the same way as Paul Johnson's answer but is more concise and robust to changes in MkResult:

import Test.QuickCheck.Property (succeeded, failed, reason)

prop a b =
  if res /= []
    then succeeded
    else failed { reason = reason }
   where
      (res, reason) = checkCode a b