Creating a culture of quality: Part 1

Before veering off rapidly into more technical topics, there's still some basics to cover off. For code samples, I’ll be using a made-up language based on JavaScript, but with a couple of syntactic tweaks where it makes the code simpler. I don’t honestly know if there’s a testing framework for JS that looks like this, but it should be fairly trivial to write one if anyone’s interested in a collaboration.

I get caught up in conversations about testing a lot these days, it’s one of those things that crops up – you know – at parties and things. It fills time between the hors d’oeuvres and the ketamine.

It seems that every time I have a conversation with someone about testing, and how to start doing it, there’s one question that is preying on their minds, to wit: “I should be unit testing, shouldn’t I? “ “I hear it’s teh awesomes!”

Yep, you should be unit testing every breakable line of code. So should I, but I don’t. It works REALLY well. For object-oriented languages in particular, Test-Driven Development directs design in immensely useful ways. Unit-testing is a vital software practice and, implemented properly, is the best way to raise the quality of your codebase, and your faith in the final product.

There are lots of other kinds of tests, though.

TDD has become a buzz-word, which is really unfortunate because while the ideas and the tools behind it are rock-solid, it’s become harder to talk to people about basic testing when they have these visions of an endless sea of unit tests. It also means that everybody puts it on their CV now and you can no longer use it as any kind of indicator. Ah well. Here’s a rundown on the various kinds of testing we practise at Huddle.

Unit tests

Unit tests are small tests which verify that a small unit of code works in the way that you expect

Let’s say that we’ve written this function

func Add ( a, b )
{
    return a + b
}

Here’s a unit test for the function.

func testOnePlusOne()
{
    Test.That(Add(1, 1)).Equals(2)
}

A unit test checks a single thing about a single piece of code. Here’s another one

func Invalid_A_Is_Error()
{
    Test.That(Add("fruit", 1))
        .CausesError("fruit is not a number")
}

This test checks that we get a sensible error back when we try to add a string and a number.

Unit tests are small, and have no side effects or hidden dependencies. That means no talking to the database, and no calls to web services, and no calling the file system. Keep It Simple and keep it fast.

You can run thousands of the little blighters really quickly, and they help to specify the behaviour of your system at a low level.

Use negative tests – tests that check for an error condition – liberally. If I’m doing TDD, my negative tests will often outnumber positive tests 3:1 (there’s that magic number again).

Specifying your edge cases in unit tests stops them coming back to haunt you at a later date.

When a unit test fails, fixing it must be the top priority for the dev team. Unit tests are, after all, specifications. If they’re broken, then so is your app. Don’t ever be tempted to live with a bunch of failing unit tests, or they’ll soon add up, and you’ll be back to square one. Square NEGATIVE one, even, because now you have all these tests that say the functionality should do X, when in fact it does Y.

Integration tests

Integration tests are larger-scale tests that reach across several components.

Sometimes, we at Huddle get tired of writing one-line functions that add two numbers together, and we get this craving to write some actual software. When we’re writing a larger component, we use integration tests.

Integration tests can have side-effects, and they check that a set of components are all working correctly together.

Here’s an integration test

func Convert_Basket_To_Sale()
{
    basket = GetTestBasket();
    sale = SaleManager.Approve(basket);
    Test.That( PaymentSystem.Verify(sale) )
            .Returns(result | result.Success == true);
}

Integration tests can talk to real databases, and call real web services. They can upload real files from real file systems, and generally do real stuff with the small chunks you previously unit tested.

Here’s another one

func Empty_Basket_Fails()
{
    basket = GetEmptyBasket();
    sale = SaleManager.Approve(basket);
    Test.That( sale.Status ).Equals(“Invalid”);

    Test.That(PaymentSystem.Verify(sale) )
            .Returns(result | result.Success == false);
 }

Integration tests should pass. If they don’t, it might be because the testing database is set up wrongly, or because the environment isn’t configured right. A failed integration test is a signal for someone to stop what they’re doing and go poke around. We’re about to start running all our integration tests on a two hour schedule, with the most important run being at 5pm. Since we tend to leave at 6 or 6:30, that gives us time to make sure that everything is working properly before we leave the office.

For both unit tests and integration tests, it’s fairly simple to use standard testing frameworks. The really difficult part here is being able to write a system that allows your tests to be simple, but that’s waaaaay out of scope for this post. System tests

System tests are end-to-end tests that check a particular task or path through the software.

When you’re working out what the functionality of your system is going to be, you should already be figuring out how to test it. If you get right to the end before realising that it’s impossible to test, you’re up the proverbial creek.

Each User Story that we write is backed by a test-plan. Developers sit down with QA to work out what the test plan for the story will be. This is really useful for working out all the little edge cases that you hadn’t thought through yet, and it means that you have a kind of to-do list when you come to write the code.

Once the test plan is written, the Story Owner and the Customer check it over to make sure that it all looks sensible and to raise any queries with QA. We’ve also started assigning a Test Owner to stories. It’s the job of the test-owner to run through the test plan when the story is complete and do a first-pass QA. We found that otherwise some members of the team do lots of testing and some (like me) do none.

You do not have to automate system tests, but it helps. You can look at tools like Watir or Selenium to drive automated tests through a web browser. It’s a hoot because you click a button and 50 browser windows pop up and all start to manically fill out forms and load customer details and click on buttons and send emails to see what breaks. I’ll be coming back to automated UI testing in a later post, so watch this space, and mail me if you have any specific questions.

For the API work, we’re using TinyGet and a little tool we wrote called TinyPost to verify system-level behaviour. Our system tests are just batch files that make a bunch of requests to the API and test all the cases that we came up with while specifying the behaviour. CURL would work just as well here, but we’re on an MS stack, and it’s easier to use Windows tools than to mess around with Cygwin.

It’s important to remember that you don’t have to write tests in the same language as your application. Learn Ruby or Python or something. Hell, script something up in Javascript – System level tests don’t have to be written by developers, either, you can find free tools that automate writing browser-based test scripts. Find an impoverished student who knows what WGET, CAT and GREP do. Whatever.

Performance and Load Tests

Performance tests verify that a given chunk of code runs in an acceptable amount of time. Load tests verify that a chunk of code can be run simultaneously by an acceptable number of users. It’s important to remember the difference. You might have an application that runs really slowly but can support 5000 concurrent users, or you might have an application that’s lightning fast until you’ve got more than three people clicking buttons at the same time.

At Huddle we’re writing our own load and performance tests. There are frameworks out there that will make the job easier, but we’ve got some specific requirements that make a custom tool easier to wield.

The important things to remember before you start these tests are:

• Work out, before you start, how fast is fast enough.
• Don’t EVER assume that you know which bits of your application are the slowest.
• Don’t EVER EVER EVER assume that you know which bits of your application are the slowest.
• Work out, before you start, how fast is fast enough.
• STOP once the code is fast enough.

When I was working on the Filing Cabinet page, back in the golden days of Iteration 2, I ran a CPU profiler against the code because it seemed a little slow. I discovered, to my amused horror, that we spent 80% of the CPU time translating the words “folder”, “you are editing this file”, etc into the user’s language. By caching these strings up front, I cut the rendering time of the page by upwards of 75%.

My first instinct would have been to tweak stored procedures, cut the number of loops, do all the OBVIOUS stuff, but no – declaring 10 or 15 variables with cached values slashed the response time. I’ve found this over and over again. Put simply, if you’re trying to speed up your application, and you don’t have real repeatable metrics on which bits are the slowest, then you’re an idiot.

You heard me right the first time: An Idiot. Trying to optimise code without a profiler is like playing darts blindfolded. Not only will you fail to score any points, but you might just put somebody’s eye out.

tl; dr: Don’t get hung up on The One True Way To Test – that can all happen later, but START WRITING AUTOMATED TESTS as soon as possible.