Tuesday, October 30, 2007

Introduction to xUnit

Over the last few months, I've been hearing more and more about the xUnit testing framework.  It's a next generation unit testing framework for .NET that is being coordinated by one of the creators of NUnit, James Newkirk.

For the most part, xUnit came about because of two reasons.  First is that the technology, practices and patterns surrounding Test Driven Development have outgrown the frameworks that support it.  Second, is that in an attempt for test frameworks to keep up with this growth, they have become bloated and filled with constructs that aren't needed.  None of the existing frameworks have truly been able to address all of these changes in a really graceful way, because they have become obligated to support the millions or billions of tests that have already been written using them.

I've tried to identify and layout as many of the primary differences between the NUnit and xUnit frameworks below, in an attempt to give a clear understanding of what it would mean for a developer to make the switch from NUnit to xUnit.  One big difference you'll notice right away is download size.  xUnit weighs in at a mere 75k download, while NUnit is around 1.3MB.  Now, NUnit has a GUI, and xUnit doesn't.  But if you ask me, I think xUnit should stick to their guns and leave it without the GUI.  ReSharper and TDD.NET do great jobs at the UI already.

Changes to Test Decoration

One big change is that some of the semantics that decorate a test have changed.  In xUnit all public methods decorated with the Fact attribute are considered members of the the test suite, similar to JUnit, and the Test attribute has been changed to the Fact attribute.  Below is a very simple test.

public class MyClassTest
{
    [Fact]
    public void TestAdd()
    {
        MyClass myClass = new MyClass();
        int result = myClass.Add(1, 2);
        Assert.Equal(4, result);
    }
}

Single Object Instance Per Method

This is something that has been discussed a lot regarding test isolation.  JUnit has been using this pattern for some time now.  A while back, James Newkirk made a blog post about how he makes all his member variables static in his NUnit tests.  Martin Fowler blogged on the subject too.  Primarily, it's purpose is test isolation.  I've made an example of a pair of tests that succeed below in xUnit, but would fail in NUnit, all because each xUnit test is run as a separate object instance.

public class MyClassTest
{
    private List<string> items = new List<string>();
 
    [Fact]
    public void TestOne()
    {
        items.Add("One");
        Assert.Equal(1, items.Count);
    }
 
    [Fact]
    public void TestTwo()
    {
        Assert.Equal(0, items.Count);
    }
}

Setup and Teardown Attributes Removed

Another hot topic along the same lines as object instantiation, the per test setup and teardown methods don't exist at all in the xUnit framework.  Again, James Newkirk has a pretty good post on it here.  The bottom line is that the test is much easier to follow.

ExpectedException Removed

The ExpectedException attribute has been removed and has been replaced with an Assert.Throws delegate function.  This allows you to wrap the specific operation that you expect to throw the exception in the delegate.  This allows you to isolate the location within the test, as opposed to expecting the exception somewhere in the test.  The later could hide other exceptions and lead to an obfuscated test result.  Below is an example.

[Fact]
public void TestOne()
{
    MyClass myClass = new MyClass();
    Assert.Throws<ArgumentNullException>(delegate { myClass.CantPassNullOperation(null); });
}

Introduction of Aspect-Like Concept

Tearing down data from a test has always been an issue, when the DB is a component of your tests.  Because of this issue, aspect-like patterns arose to setup and rollback transactions to ensure that tests did not step on each other.  One implementation of this was XtUnit by Roy Osherove.  xUnit makes rolling your own aspects very easy.  Below is an example with console output tracing.

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TracingAttribute : BeforeAfterTestAttribute
{
    public override void Before(MethodInfo test)
    {
        Console.WriteLine("Before Test : {0}", test.Name);
    }
 
    public override void After(MethodInfo test)
    {
        Console.WriteLine("After Test : {0}", test.Name);
    }
}
 
public class MyClassTest
{
    [Fact, Tracing]
    public void MyTest()
    {
        Console.WriteLine("During Test");
    }
}

Extensibility

In addition to these new features, xUnit allows for quite a bit of extensibility.  You can extend the framework from the test class level, all the way down to the assert level. 

At the class level, you can override how the test is run completely.  xUnit ships with an example of this in the xUnit.extensibility assembly, that tags the test class as an NUnit class.  This allows you to use the xUnit runner to run NUnit tests.

At the method level, you can decorate your class with an extension of the Fact attribute, and control how that individual test is executed.

And finally, at the assertion level, you can control the comparer that is used when running comparison tests (Equal, NotEqual).

Test Runners

As of this point, the only test runner shipped with xUnit is console based.  However, there are ways for you to configure both TestDriven.NET and Resharper to run xUnit tests.  xUnit is currently at Beta 2, and can be downloaded here.

License

The xUnit project is licensed under the Microsoft Permissive License, which allows redistribution, but not without all copyright, patent, trademark and attribution being distributed with it.