Weirdness with cabal, HTF, and HUnit assertions

Question

So I'm trying to use HTF to run some HUnit-style assertions

% cat tests/TestDemo.hs
{-# OPTIONS_GHC -Wall -F -pgmF htfpp #-}
module Main where
import Test.Framework
import Test.HUnit.Base ((@?=))
import System.Environment (getArgs)

-- just run some tests
main :: IO ()
main = getArgs >>= flip runTestWithArgs Main.allHTFTests

-- all these tests should fail
test_fail_int1 :: Assertion
test_fail_int1 = (0::Int) @?= (1::Int)

test_fail_bool1 :: Assertion
test_fail_bool1 = True @?= False

test_fail_string1 :: Assertion
test_fail_string1 = "0" @?= "1"

test_fail_int2 :: Assertion
test_fail_int2 = [0::Int] @?= [1::Int]

test_fail_string2 :: Assertion
test_fail_string2 = "true" @?= "false"

test_fail_bool2 :: Assertion
test_fail_bool2 = [True] @?= [False]

And when I use ghc --make, it seems to work correctly.

% ghc --make tests/TestDemo.hs
[1 of 1] Compiling Main             ( tests/TestDemo.hs, tests/TestDemo.o )
Linking tests/TestDemo ...
% tests/TestDemoA
...
* Tests:    6
* Passed:   0
* Failures: 6
* Errors:   0

Failures:
  * Main:fail_int1 (tests/TestDemo.hs:9)
  * Main:fail_bool1 (tests/TestDemo.hs:12)
  * Main:fail_string1 (tests/TestDemo.hs:15)
  * Main:fail_int2 (tests/TestDemo.hs:19)
  * Main:fail_string2 (tests/TestDemo.hs:22)
  * Main:fail_bool2 (tests/TestDemo.hs:25)

But when I use cabal to build it, not all the tests that should fail, fail.

% cat Demo.cabal
...
executable test-demo
  build-depends: base >= 4, HUnit, HTF
  main-is: TestDemo.hs
  hs-source-dirs: tests
% cabal configure
Resolving dependencies...
Configuring Demo-0.0.0...
% cabal build
Preprocessing executables for Demo-0.0.0...
Building Demo-0.0.0...
[1 of 1] Compiling Main             ( tests/TestDemo.hs, dist/build/test-demo/test-demo-tmp/Main.o )
Linking dist/build/test-demo/test-demo ...
% dist/build/test-demo/test-demo
...
* Tests:    6
* Passed:   3
* Failures: 3
* Errors:   0

Failures:
  * Main:fail_int2 (tests/TestDemo.hs:23)
  * Main:fail_string2 (tests/TestDemo.hs:26)
  * Main:fail_bool2 (tests/TestDemo.hs:29)

What's going wrong and how can I fix it?

Answer 1

This is a bug in certain versions of GHC, related to the optimizer removing throwing IO exceptions in some cases. It is very common with HUnit code, if optimizations are enabled. And cabal sets -O1 by default, which enables the bug.

Upgrading to GHC 7 fixes it (not really recommended until libraries have caught up with it, IE a haskell platform release is made for GHC 7).

You can also put -O0 in the compiler options in the .cabal file stanza related to your test executable. This is what I've done for my test code, until I am ready to move my project to GHC 7.