Although the XP community has been one of the main driving forces behind unit testing in the last few years, it is not a new concept. Indeed, the seminal book “The Mythical Man Month” (Frederick P. Brooks Jr) from 1975 describes testing as an important and intrinsic part of the development of components and systems. Unit testing can and should be a part of any software development program, regardless of the methodology in use. Even if the project management team doesn’t require the use of unit testing, it is a valuable tool for the developer.
There have been a number of great articles recently on unit testing (see Kevin Jones’ article), with examples of how to write unit tests and how to use some of the common component testing frameworks. My main problem when I started out using unit testing, however, was not how to use it from a purely technical perspective, but rather how to integrate it into my project lifecycle. I have now been through a number of iterations, so I thought I would share my “Top 10” tips for integrating unit testing into your development process.
1. Never ship your tests
Code for unit testing should by definition never be used in production code, therefore it is not needed in your product when you ship it. The code won’t run without a test framework, however, so what harm can there be if a few uncalled functions end up in your binaries – they won’t take up much space, right? Apart from combating software bloat, here are a few more reasons why test code in a released product may not be as benign as it first seems. The first should be the mantra of every software professional – security. You may view your unit tests as being “harmless”, but are you sure you didn’t hardcode the “sa” or “admin” password for your corporate database in a test? Have you audited your test code thoroughly to ensure that executing it can’t allow a user to do something they shouldn’t be able to do? Is the test data you’ve embedded in your code “clean” if a customer was to read it? Does your test code compromise any licensing features in your code? Another reason for not including test code is that it can give anyone looking at your software insights into its inner workings that you may not want them to have. Even if your code is not commercially sensitive, do you really want others to be able to see how you test your code? Especially with languages like Java and C#, de-compilers are common and your tests will tell others a lot about both your code and your development methodology. Last but not least, imagine the embarrassment you would feel if a customer called up and told you they had run the unit tests in your application and they had failed!Depending on the environment you are using, there are a number of different ways to prevent test code escaping into the wild. Conditional compilation can be used to prevent unit tests from being present in release builds. However, there are good reasons why you would want to run unit tests on your releases so this may be flawed. Extra project configurations “Release with Unit Tests” and “Release” can be used to work round this. Building a separate test module is another solution, a separate library or assembly containing nothing but unit tests. This has the advantage that the tests are all together in one place for the test tool, and that tests can easily be run on the very same binary files that are released to customers. There is also the advantage that as your unit tests improve, you can potentially combine a “new” test module with old release binaries in order to see whether old releases comply with the latest test specifications.
2. Start at the start and keep going
Some project lifecycle methodologies require that we write unit tests before any other part of the code. In many cases, this is simply not practical. It is seldom that I get to work on “blank slate” projects where there is no existing codebase to work from – so at least some of the code is going to exist before the tests. That said, when starting a new piece of work you should set up your project for unit testing early on, and write tests for any existing code that you are likely to modify. Apart from the benefits of having unit tests in place, writing tests for existing code is a really great way to figure out what it does and to avoid creating bugs when you use or modify it. When you do have to make changes to a module or create a new one, then planning the unit tests up-front is a good way of capturing your understanding of what is required. In some cases, you can actually write the code for the test, but unless your methodology specifically requires this, I find that creating the framework for the new tests and putting in comments (marked “TODO”) describing the process in detail is usually enough to focus the mind.As important as writing tests at the start of a project is maintaining those tests. Even when a project is “finished” and in the maintenance phase, it is still important to update the unit tests to catch “bugs” that are observed in the field – remember, any bug found in the field is a bug in the unit tests as well as a bug in the code. Doing this will ensure that the bug is not re-introduced as part of a future “fix”, or when code is re-factored. Even more importantly, if the code is adopted for a new development the tests will already be in place and up-to-date.
The first couple of projects I did with unit testing, I ignored these rules. I wrote my tests all together near the end of the project, I wrote them after my code and my main motivation was to tick something off my to do list. I ended up with weak tests that missed bugs. You will only get the real benefits unit testing has to offer if you integrate them with your project from the start, and keep them up to date as new test cases come to light.
3. Test code is a first class citizen
Having read the previous tip, I hope you have realised that you should be thinking about your unit tests any time you are working on your code. It shouldn’t come as a surprise, therefore, that you should treat your unit test code exactly as you do the rest of your code. This means that coding standards, documentation, peer review, in fact anything that you do with your production code, should be applied to unit test code too.Another very important aspect of this is that Test Code should be maintained in source control alongside production code. The layout of the source control repository should make it straightforward to check tests in and out at the same time as the modules/classes they refer to, with the same comments and labels. For unit tests to be really useful, the test code must be a first class citizen in your project, and must be afforded all the same attention that production code is.
4. Unit tests must be portable
It is all too easy when writing test code to unconsciously make assumptions about the environment it is running in. This might take the form of hard-coding the directory of a test data file, or assuming that a certain server is available. Some of these assumptions may be valid – if the only people to ever run a test will be in your office and will have access to the same servers as you then that is unlikely to cause a problem. However, it is important to ensure that any assumptions that tests make about their environment are justified, if only because your colleagues won’t thank you for checking in tests that fail on their machines.I am personally guilty of one of the worst cases of this crime that I have seen. I was writing a system that included some software protection facilities, one of which relied on the Windows Serial Number of the machine it was installed on. Some time later, I was building and testing this code on a new laptop, and I couldn’t figure out why one specific test just kept failing even though all the others were OK. Eventually I found that I’d hard-coded a test licence string that was based on the windows serial number of my main development PC. The solution in this case was to modify the test so that it had a list of licences and used the one matching the machine it was running on. If no licence was found for that machine, it threw an exception making it clear exactly what had gone wrong, where and why.
In most cases the dependency can be eliminated with a little thought about the structure of the tests. However, solutions like the one I used in the above example can save a lot of trouble where a dependency is inevitable. Another trick that you can use to avoid introducing dependencies is specifying parameters in a config file that is maintained per-machine or per-location.
Where a test has a dependency on a file on the test machine, one way to avoid having to propagate test data separately is to build it into the test library/DLL as a resource or as static data, then write it out to disk as part of the routine that prepares the tests for running. This way the file is always there, in the right place, and compatible with the version of the tests in use.
5. Integrate test execution
If you make it easy for yourself and the project team to run unit tests, then it is much more likely that they will actually be used. Tests that aren’t used are no use, and test driven development methodologies can require testing after every compile – so make it easy. Integrating your test tool with your development environment is one way to do this (if they are not already integrated). Usually it is possible to set up library projects so that they use a named executable to load the library when you run the project – try pointing this at your test execution engine. If you wish to keep your desktop free of clutter, you may wish to use a command line version of the test tool to display test results within the development environment. In either case it is important to make sure that it is possible to run your tests via the debugger. That way when you do have a test that fails you can at least find out why.For many environments other people have already figured out how to integrate test tools, and often there are add-ins or scripts that make the process painless. For example, if you use Microsoft Visual Studio then the “TestDriven.net” tool will provide seamless integration for nUnit and various other test frameworks
6. Automate test execution
If for some reason you find that tests are not being executed as regularly as they should (it’s easy to get lazy, isn’t it?), one way to resolve that is to automate the running of unit tests. One way of running tests automatically is to invoke them from the build script. This may only be practical if the tests are relatively quick to run, and you can save developers from having to run lots of irrelevant tests on each build by only implementing it for release builds. That way, you can’t release code that hasn’t been unit-tested by mistake. Another option is to have a scheduled event that builds the source tree from the repository each night, runs the unit tests, and emails any failures to the whole team. With this approach any code that has been checked in without testing will be noticed very quickly and the appropriate action can be taken.7. Relate tests to requirements
If there are any formally expressed requirements for the unit under test, or any system requirements that can be applied to it, then it makes sense to formally relate the unit tests to the requirements. Usually it will take a number of separate unit tests to completely test a single requirement from the specification. Relating tests to requirements has a number of benefits:- If unit tests are closely related to requirements then you can test whether the requirements have been met simply by running the unit tests.
- It gives the tests a context – a developer looking at each test can see why it is important and how it fits into the system specification.
- When the requirements change, it is relatively easy to see which tests need to be changed to reflect this.
8. Use test categories
On a large project, running all the unit tests could take a very long time and some tests may require some sort of user intervention. These factors can make testing a time-consuming business, so often it is desirable to flag tests that are slow or require interaction so that they are only run when explicitly requested. Most test frameworks provide facilities for categorizing tests in this way, but it is up to the developer to ensure that categories for the project are well defined and consistently applied.9. Test “what” not “how”
When writing a set of tests it is very important that you test what the component does, and not how it does it. This is because an important use of unit tests is to enable the implementation of a unit to be changed (perhaps by re-factoring). As an example of what this would mean in practice, consider a function that is required to return a list of files in a directory. In our test code we have an array of filenames:{ “one”, “two”, “three” }
…and use these to create files in a test directory. We then check that the returned list contains three entries, and matches our array.
This test may or may not succeed, because it imposes an additional requirement on our function that was not in the original specification. It requires that the filenames be returned in a specific order. In order for this test to be correct it must check that the output list contains each of the required names and no other spurious names, but it must not create a new requirement on the order of the names. This requirement is often taken to mean that unit tests should be developed with no knowledge of the implementation of the units being tested, however, this is simply not the case. One example where looking at the underlying code is important is using code coverage analysis to ensure that unit test cases cover all possible routes through the implementation. This can result in adding new tests to the system – but they should still be testing what the unit does under a given set of circumstances rather than how it does it!
10. Don’t forget other forms of test
Unit testing is a very important form of testing, but it should not be used as a substitute for other forms of test! Testing all the pieces of a system individually does not prove that they will work in the intended way when combined into a complete system. Unit testing typically provides little or no information on:- User interface code and logic behaviour
- System integration code behaviour
- Performance
- Compatibility with different hardware and software platforms
- Usability
- Scalability
Conclusions
These are my personal “Top 10 Tips” for integrating unit test with your project lifecycle. They are all responses to problems that I encountered when I first started to bring unit testing to bear on real projects, and I hope that they will help you avoid some of the problems I experienced. If you are looking for more resources on unit testing, testdriven.com has some very active forums, and the websites for the tools mentioned in the article all have links to resource and community sites where you can pick up great tips and learn from the experience of others.Ian Stevenson has been developing Windows software professionally for almost 10 years, in areas ranging from WDM device drivers through to rapid-prototyping of enterprise systems. Ian currently works as a consultant for The Generics Group, and can be contacted at [email protected].
Unit testing web resources |
Comments