Specifically, we’ll be looking at the integrated support for:
- unit testing
- code coverage
- code analysis
- refactoring
Figure 1: Visual Studio 2005 Team System
A quick introduction to Test-Driven Development
The practice of Test-driven Development (TDD) has been around for several years now, tracing its origins back to work from the agile programming community, and is based on a simple three-step process:- You write a failing unit test
- You develop the production code only until the test passes and no more
- You refactor the code to remove any duplication and to make the code as simple and easy to understand as possible, remembering, of course, to re-run all of your unit tests to make sure that the refactoring didn’t break anything
There are also a number of ‘fringe’ benefits when using TDD, such as:
- having more maintainable code as the refactoring tends towards making code simpler. In fact refactoring to make things simpler is not just a good idea in TDD, it’s the mantra.
- having more documentation for your code (the unit tests)
- letting you examine your code from the perspective of a developer that has to actually use your classes.
- allowing you to track the progress on a project by the number of passing unit tests that exist
Unit testing with Visual Studio 2005 Team Developer
Perhaps the most interesting way to examine the concepts of TDD is to see an example in action. Let’s consider a very simple Math class, which our design currently states must support a single Add() method for adding two integers. And if you’re thinking that a Math class which has only an Add() method seems a little ‘lightweight’, remember that a core philosophy in extreme programming is to only implement the features you need for the current project, not those for every possible future use.In a perfect TDD world, we would write the unit test for this class and method first. However, a more pragmatic approach when using Visual Studio 2005 Team Developer, and certainly one that you might use for more complex projects, is to get the tool to write the shell of the code and its corresponding test for us.
Figure 2 shows the Math class after it has been created by the Visual Studio 2005 class designer. Using the class designer is probably overkill for our simple scenario, but I’ve used it to highlight the tight integration of the different parts of Visual Studio 2005 that come together to support our coding.
Figure 2: The Math Class Diagram
The class designer then automatically generates the code:
public class Math
{
public static int Add( int i1, int i2 )
{
throw new System.NotImplementedException();
}
}
The Unit Test Generation Tool
We can now use the Unit Test Generation tool to create our unit test code. This is accessed from the View | Quality Tools | Create Tests… menu option, and is shown in Figure 3.
Figure 3: The Unit Test Generation Tool
The Unit Test Generation tool can be used to create and maintain tests for all of the types in a project. It generates the following test code for the Math class, with comments removed:
[TestClass()]
public class MathTest
{
[TestInitialize()]
public void Initialize()
{}
[TestCleanup()]
public void Cleanup()
{}
private TestContext
m_testContext = null;
public TestContext TestContext
{
get { return m_testContext; }
set { m_testContext = value; }
}
[TestMethod()]
public void AddTest()
{
int i1 = 0;
int i2 = 0;
int expected = 0;
int actual;
actual = Math.Add( i1, i2 );
Assert.Inconclusive(
“Verify the correctness of
this test method” );
}
}
Let’s look at this generated code now.
The test class
Visual Studio 2005 Team Developer will place all of the unit tests for a single type inside a single test class, which has the name as the type appended with the word Test (so that’s MathTest in our example). The test class name is actually irrelevant, as the more important item is the TestClassAttribute that is applied. This attribute resides in the Microsoft.VisualStudio.QualityTools.UnitTesting.Framework namespace, along with the other types and attributes that I am about to discuss.The idea, of course, is that all of the unit tests that we write for the Math class should reside in the MathTest test class. In the real world, your tests will probably be a little more complex than just checking to see whether two numbers are being added correctly. Consequently, the test class contains two methods, Initialize() and Cleanup() that will be run before and after each test method respectively. You can therefore use these to implement common initialisation and clean up code that would be used by all of the tests, such as retrieving data from a database or disposing of a GDI+ object. As you would expect, you can add and use instance members in the test class to hold any objects that are required, as there is no way of passing parameters into either the test method or the clean up method. Note that these methods are decorated with the TestInitializeAttribute and TestCleanup attributes.
It is important to remember that the initialisation and clean up methods are called once per test, as each test is run independently from any other tests and in any order. Therefore, your tests should make no assumption that any other test has been run or will be run subsequently; you’re unit testing, not sub-system testing.
The test method
The actual unit test is in the method called AddTest() (spot the naming pattern), which is, more importantly, decorated with the TestMethodAttribute. Each unit test that you write will have its own method. An important thing to note, here, is that the wizard can generate a set of test code, but it’s unsure whether the code that it generates is valid. Consequently, the final statement in the test uses the Inconclusive() method of the Assert class to flag the fact that the test code needs to be examined. Assert is used to perform all of the evaluations to determine the success or failure of the unit test, and it contains methods to perform equality and inequality checks, including reference equality, tests for false or null references, and so on.In our example, it takes but a moment to modify the test code to create the failing unit test:
[TestMethod()]
public void AddTest()
{
int i1 = 2;
int i2 = 3;
int expected = 5;
int actual;
actual = Math.Add( i1, i2);
Assert.AreEqual<int>(
expected, actual);
}
In this example, I’ve chosen to use the generic version of the Assert’s AreEqual() method to perform integer comparison. As you can see, your unit test code shouldn’t contain any UI involvement and should be as simple as possible. At this point, the unit test is also a failing unit test, as the Add() method currently throws a NotImplementedException, as shown in the first listing.
Let’s look at how to run the tests.
Test View and Test Explorer
The most common way to view and run your unit tests is through Test View or Test Explorer. These let you categorize, sort and select tests to be run, although in many cases you’ll want to run the entire set of tests for a project (this comes back to making sure that you catch any breaking changes as you code).Figure 4 shows these two views in action, along with the results of running our failing tests in the Test Results tab in the bottom left hand corner.
Figure 4: Test View
Tests can be queued and run on a target unit test machine if necessary. In fact, you will have to configure the run settings for the tests before you run them for the first time. We’ll take a look at how you do this a little later, when we examine how to add code coverage analysis to your unit testing.
You can navigate to the source code for a test from the Test Results, Test View or Test Explorer, and you can see detailed output and error messages for any failing tests.
In our Add() method example, now that I’ve run the test and seen it fail, it’s simply a matter of implementing the addition code and re-running the test to make sure that it now works.
Tests and exceptions
Of course, it doesn’t take long until the design changes. Consider the following additional requirement being added to the Add() method; it has to throw an exception if the result of adding the two integers is too large to be held in an integer (remembering that, by default, C# performs unchecked mathematical operations).Following our practice of TDD means that we need a new unit test to cover this scenario. The trick here, of course, is that the unit test will be successful if an exception is throw, and will fail if there is no exception. Rather than have to add your own try and catch code, the testing framework lets you mark a unit test as expecting an exception with the ExpectedExceptionAttribute:
[TestMethod()]
[ExpectedException(typeof(System.OverflowException) )]
public void AddWithOverflowExceptionTest()
{
int i1 = System.Int32.MaxValue;
int i2 = 1;
Math.Add( i1, i2 );
}
Having determined that the AddWithOverflowExceptionTest() unit test fails, a bit of quick re-implementation of the Add() method results in:
public static int Add(int i1, int i2)
{
int result = 0;
checked {
result = i1 + i2;
}
return result;
}
At this point we would re-run both unit tests to make sure that nothing has been broken.
Debugging your tests
There is one minor irritant with the current preview versions of Visual Studio 2005: you can’t trivially debug a test. This is an issue that should be resolved when the product ships, but there is a relatively straightforward workaround. MSTEST.EXE, which can be found in the \Common7\IDE subfolder of the Visual Studio 2005 installation, is a command line tool that you can use to run your unit tests. Its most important command line argument is used to specify the assembly that contains the tests. In our example, we’d run:mstest /testcontainer:qa.math.dllConsequently, debugging the unit tests simply involves setting MSTEST.EXE as the debug target for the project with the appropriate command line parameter established, an example of which is shown in Figure 5.
Figure 5: Debugging unit tests
Once you’ve set this up, you will be able to debug your code through your unit tests, which can be quite an attractive proposition if the code is relatively complex.
Code coverage
At this point we can all sit back and feel quite happy about our new, ‘bug-free’ coding habit. However, there’s one small voice whispering (well, nagging actually) at the back of my mind: “Are you sure that your unit tests are actually testing all of the code?”An important metric when determining the quality of code is how much of it has been tested; this is known as code coverage analysis, and is something that developers should be thinking about every time they come to check in code. One of the things that developers who use Visual Studio 2005 Team System will need to get used to is that a configurable policy can be used to enforce a minimum percentage level of the code that has been tested. How will they know whether their tests are meeting this requirement? Fortunately, Visual Studio will help developers here by performing a code coverage analysis when they run their tests.
Again, I will use a simple example to demonstrate this in action. In this case, I’ve added a new method (and a couple of unit tests) to the Math class to support calculation of a square root using Newton’s algorithm, which returns a double to the requested number of decimal points. The code for the SquareRoot() method is:
public static double SquareRoot(
double i, int numDecimalPoints )
{
if( i <= 0.0d )
{
throw new
ArgumentOutOfRangeException(
i, “Must be greater than 0”
);
}
double approximation = i / 2;
double delta = 1e-10;
while( Abs(approximation *
approximation - i) > delta )
{
approximation = (approximation
+ i/approximation) / 2.0;
}
return Round( approximation,
numDecimalPoints );
}
Note that the code for the two private methods, Abs() and Round(), are not shown for clarity.
A quick look at the unit test for this method shows a range of values being tested:
[TestMethod]
public void SquareRootTest()
{
Assert.AreEqual<double>(
1.0, Math.SquareRoot( 1, 1 ) );
Assert.AreEqual<double>(
2.0, Math.SquareRoot( 4, 1 ) );
Assert.AreEqual<double>(
3.0, Math.SquareRoot( 9, 1 ) );
Assert.AreEqual<double>(
4.0, Math.SquareRoot( 16, 1 ));
Assert.AreEqual<double>(
5.0, Math.SquareRoot( 25, 1 ));
Assert.AreEqual<double>(
16.0, Math.SquareRoot(256, 1));
Assert.AreEqual<double>( 100.0,
Math.SquareRoot( 10000, 1 ) );
Assert.AreEqual<double>( 2.8284,
Math.SquareRoot( 8, 4 ) );
}
Enabling code coverage analysis
The big question is whether this test covers all of the code paths inside the SquareRoot() method. As briefly mentioned before, you can configure various settings when you set up your unit tests. Figure 6 shows the Code Coverage page of the configuration tool. All you need to do is enable code coverage and add the requisite binaries to be instrumented. Then simply run the unit tests as before.
Figure 6: Enabling code coverage
It’s extremely unlikely that you will want to run code coverage analysis all the time, as it has a significant impact on the length of time it takes for each instrumented test to run.
Viewing the results
With the tests completed, you can quickly view the results. The Code Coverage Results window (shown in the bottom of Figure 7) provides the hard information needed to produce reports on code coverage. There is also a very powerful visual display available in the normal code window. This involves highlighting lines of code that have been covered by the tests in green, whilst those that have been missed are shown in red. Again, this is also shown in Figure 7.
Figure 7: Code coverage results
Our example highlights one of the most common testing errors: failure to test error handling code. Resolving the problem simply involves adding a second unit test crafted to ensure that the exception is generated correctly.
Check in, build and test
Unit tests and code coverage analysis are an incredibly important aspect of the check in and build process. Hopefully you will be performing daily builds, as this provides an excellent measure of the progress and stability of the project. Running your unit tests and performing code coverage analysis as part of that daily build is an essential factor in monitoring code quality and code churn (how much of the code is changing). Visual Studio 2005 Team System lets you produce reports for observing these metrics, so make sure that these are integrated into your project’s reporting information.Code analysis
The combined presence of unit testing and code coverage is certainly of great benefit in reducing the bug count, but code quality is much more than just having zero defects. Having good code maintainability and code structure is equally important. Many .NET developers use static code analysis tools, such as FxCop, to help ensure that their code is fundamentally well structured. With Visual Studio 2005, FxCop is integrated into the IDE and the build process. As ever, you can easily configure the rules that FxCop will follow, as shown in Figure 8.
Figure 8: Configuring FxCop rules
As you build, any FxCop warnings or errors are displayed in the Task List, along with the usual helpful hints on how to fix the problem.
Of course, C++ developers also have the luxury of using PREfast to check for possible coding problems.
Refactoring
Part of the process of TDD is refactoring of code. This typically involves everything from renaming a badly named method or field through to extracting a block of code and turning it into a method, in turn replacing other instances of that code block with a method call. Well, there is some good news (and some bad) on the refactoring front with Visual Studio 2005. The good news is that Visual C# gets considerable refactoring support, an example of which is its ability to generate a method from a block of code, as shown in Figure 9.
Figure 9: Refactoring code to extract a new method
The bad news, I’m afraid, is that Visual Basic developers are limited to simple renaming of items (no extracting of code to form procedures) and Visual C++ developers don’t get any support at all! Of course, this doesn’t mean that Visual Basic and C++ developers can’t perform refactoring, it just means that they have to do more of the work manually.
Conclusion
Test-driven development has very much joined the mainstream as a practical and effective approach when writing code, and we have taken a high level view of in this article. Currently, we tend to use tools like NUnit (from www.nunit.org), FxCop and a range of other third party products within custom designed build processes to provide a similar experience to what we will get with Visual Studio 2005 Team System.And that’s really what distinguishes Visual Studio 2005 Team System; these new tools are all fully integrated and seamless in operation. This makes them accessible enough so that they are convenient and natural to work with. Consequently, it becomes a pleasure to write test code, which is not something that I thought I would ever say!
Dave Wheeler is a Principal Technologist with QA, a Microsoft Gold Partner for Learning Solutions in the UK. He specialises in .NET application development and design.
Resources
You can find out more about Visual Studio 2005 Team System at MSDN. Note that this article was based on the Community Technology Preview version of Visual Studio 2005 Team System that was made available as part of the Beta 1 refresh of Visual Studio 2005.Good resources for further information on how you can currently work with test-driven development on .NET can be found at www.nunit.org and www.testdriven.com.
Comments