How to unit test C# code that's expected to fail compilation for negative testing? [closed]

Question

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.

Closed 8 years ago.

We're meant to generate a string output that's required to adhere to a particular set of syntactical rules. I created an object model in order to enforce that syntax via C#'s strong-typing with the intent to prevent the possibility of generating invalid output.

I can create positive tests, i.e. valid C# generating valid output. What I'm unable to do is run negative tests, i.e. ensuring that attempting to generate invalid output will throw errors at compilation.

Explicit example:

namespace Abstract
{
    public interface FooType { }
    public interface FooString : FooType { }
}

public interface Integer : Abstract.FooType { }    
public interface SingleLine : Abstract.FooString { }
public interface MultiLine : Abstract.FooString { }

public class Bar<T>
    where T : Abstract.FooType
{
    public Bar(string s) {
        // do stuff with s and T, where T is SingleLine or MultiLine
    }

    public Bar(int i) {
        // do stuff with i and T, where T is Integer
    }
}

public static class Foo
{
    public static Bar<T> Bar<T>(int i) where T : Integer {
        return new Bar<T>(i);
    }

    public static Bar<SingleLine> Bar(string s) {
        return new Bar<SingleLine>(s);
    }

    public static Bar<T> Bar<T>(string s) where T : Abstract.FooString {
        return new Bar<T>(s);
    }
}

All of that just so I can do:

Foo.Bar<SingleLine>("some string");  // ok
Foo.Bar("another string");           // ok
Foo.Bar<MultiLine>("more\nstrings"); // still ok
Foo.Bar<Integer>(24)                 // also ok

// How to test these lines for compilation failure?
Foo.Bar<Integer>("no good");
Foo.Bar<MultiLine>(-1);

In case it matters, I'm using VS2012 Express for Desktop.

Answer 1

Sure you can, just craft the code snippet that you want to fail to compile (as a string, or load a .cs file into a string) and invoke the C# compiler on it using the CodeDom. Your test then just needs to check that the compiler failed, and if you like check the line number error message etc... is correct.

Of course this is a fair amount of effort - you need to evaluate how much gain this is really going to get you. If you are developing some sort of API that other developers are going to use, and this is an important feature that may well inadvertantly break in the future through some subtle change then you may want to unit test this. Otherwise this is probably going to be a lot of effort for little reward (IMO).

Answer 2

I'd be highly suspect of code that did this. But, if you wanted to create a unit test to make sure someone didn't change code to allow a class to work in a certain way (i.e. stop having a compile error when used in a certain way, you could use the CodeDOM in this way:

CSharpCodeProvider codeProvider = new CSharpCodeProvider();
ICodeCompiler icc = codeProvider.CreateCompiler();
CompilerParameters parameters = new CompilerParameters();
parameters.ReferencedAssemblies.Add("mydll.dll");

parameters.GenerateExecutable = false;

CompilerResults results = 
    icc.CompileAssemblyFromSource(parameters, 
        String.Format(@"using System;

namespace Testing
{{
    class Program
    {{
        static void Main(string[] args)
        {{
            {0}
            Console.ReadLine();
        }}
    }}
}}
", "Foo.Bar<Integer>("no good");"));
Assert.AreNotEqual(0, results.Errors.Count);

You basically create a provider, tell it you want to reference a particular DLL (presumably where Foo is) then create the code (text--note I'm doubling up the curly brackets because they're delimiters in String.Format) you want to test, wrapped in a class (including Main if you're generating an exe) then compile the text. You can verify that an error occurred with the Errors collection.

Answer 3

Roslyn will let you do stuff like this. It will probably be built in to .net 5, or C# 6 or something. It's essentially equivalent to code dom though.

Answer 4

As you probably noticed, for a unit test to run, its code will have to be compiled succesfully, so it's a bit a chicken-egg problem.

You can solve this by calling the API using reflection. The CLR will throw an exception if you are calling a type in a way that is not compatible (for instance when you used type constraints).

Another option is to use Code Dom (or just trigger the csc.exe) using a piece of text in your unit test. You can make your test pass by letting the compilation fail.