The test, code, refactor cycle
The proposition of XP is that sharp tools and good people following a certain set of well-established development practices, all taken to their extremes, will reliably deliver quality products on time and with the features that add most value to your business. Taking practices to extremes means that if testing is a good idea then perform it all the time, if design is advantageous then make it a routine part of everyone's day, if code review is desirable then do it constantly. To some people, XP is a dangerous doctrine, but to others it's simply a lot of common sense and good practice woven into a workable framework. In this article we will look at how an XP team works, and discuss the pros and cons of the approach, so you can make an informed decision whether it's worth considering for your next development project.
The XP team is a tightly knit group of people, normally working in the same office. The team is organised into software developers who know 'how' to program, and customers who know 'what' to program. The developers program together in pairs and are expected to undertake whatever job needs doing. In a typical day, they will perform all the roles of analyst, designer, coder, tester, build co-ordinator, and system integrator.
Customers are seconded onto the team from the business and usually learn XP techniques on the job. There is at least one customer working full time with the developers to ensure business requirements are clearly and continuously communicated to the people charged with implementing them. The manager's role (if there is one) is that of a facilitator, co-ordinator and coach, rather than a boss with a big stick.
Although experience and talent are valued in XP, it is more important to have team players who are committed to the success of the project, and bring with them enthusiasm, flexibility, and a willingness to learn - in other words, 'good' people.
While you don't need to buy any expensive new tools to practice XP, you do need to have tools that maximise the team's development effort. The nature of XP means that you rebuild and test the product many times a day, so any inefficiency in these areas will be greatly magnified once you start work. It is worthwhile looking at anything that will take a few seconds off your build time or reduce the testing effort. Although some form of automated test tool is essential for both unit and functional testing, don't think that super-fast machines and expensive compilers are the only way to gain efficiency. Often, bigger gains are to be had simply by re-organising the way you use your existing tools. It is not the cost of the tool that counts, but how 'sharply' it is used.
A Day in the Life of an XP developer
Your day as a developer in an XP team might start at eight o'clock with a brief stand-up meeting so everyone can say what they did yesterday and set out their plans for the day ahead. Good communication is essential to the functioning of the team, and these morning meetings are just one of the ways that this goal is achieved. The meeting is also an opportunity for the team to review progress on the project as a whole. Team members take responsibility for optimising their performance by setting metrics, monitoring them, and then making any adjustments that might be necessary. These metrics are displayed on big visible charts fixed to the office walls and contain things like a checklist of the tasks for the current iteration, or the daily percentage of office time spent programming. People take turns in taking the role of project 'tracker', to record the data needed to produce these metrics and then update the charts. The XP team is highly self-disciplined.
After the meeting you pair up with another developer, who we'll call Fred, and pick up the next task card from the stack that you committed to completing for the next release of the product. Each of these tasks corresponds to a simple 'story' written by the customer on an index card to describe something that the product must do. This customer is sitting at a nearby desk, so if necessary the three of you can talk over the requirement before starting work on its implementation. Rather than any architectural model, you are guided by a shared 'metaphor' that captures the nature of the product in terms of a high-level image summing up how things should work. In your project the metaphor is an investment bank's trading desk.
Test Driven Development
Although you don't produce any formal design models, you and Fred sketch out an object interaction diagram to help you both understand an area of the task that is particularly complex. In XP, no documentation is required beyond the code itself, not even comments. You are following a style of programming called test-driven development (TDD). This means that before writing a single line of code to implement your task you must first write a test.
A macro-based tool provides the framework for your tests, which are written in C++ and statically linked to your product when certain flags are set in your Make file. At first the test fails to compile as the code it is testing has not yet been written. You then write just enough code to ensure that the build doesn't break while making sure that the test will fail; a function that simply returns -1. Fred runs the test and sure enough it fails - great, you're getting somewhere! You fix the function so that it passes the test by adding only the least possible amount of code; you hard-code a return value of zero. The function takes shape as you and Fred iterate though the cycle of adding a test and then adding the code needed to pass the test. All the time you are doing the simplest thing possible to make it all work.
The Simplest Possible Thing
Every time the tests run successfully, you and Fred make a point of refactoring, which involves looking for a way to improve the code by making it easier to understand, more flexible, and better quality. However, in the first instance you always code the simplest solution and only look for a more general solution when the issue is encountered again. You never duplicate your own or anyone else's code. In XP there is collective ownership of the entire code base, and everyone is required to refactor whenever the opportunity arises. This approach creates few problems, because everyone is familiar with all areas of the code, and as the product is rebuilt and tested many times each day the work of each programming pair is subject to continuous integration and validation by TDD.
The two of you work hard, and within a few hours you've almost completed the task. It's surprising that even when working with a less experienced programmer like Fred, pair-programming often results in better code, written in less time, with fewer bugs. He also acts as your conscience, reminding you of the coding standards agreed by the team, which you rather grudgingly accept as being for the common good. Pair programming is a very social activity and so following a satisfying morning's work you take lunch together.
After lunch you check your work into SourceSafe (a tool that provides version control for the product's source code files) and start to create a new build of the product. Unfortunately, there's a problem due to a conflict with someone else's code. Although there is no documentation or comments it is not difficult to find out what is going on by looking at the code and the associated tests. Fred checks out the offending file, corrects the problem, and then runs the unit tests to check for any unexpected consequences of the code change. All the tests pass, so he checks-in the file and completes the build. You successfully run the integration tests against this build to complete the work for your task.
The customer, hearing you've finished the task, starts-up the automated test tool and runs the functional test that she and the team's test expert wrote that morning. The product now does what the task specified it should do, so a tick is put in the box on the wall chart for all to see. You record the time you have spent working on the task so that in the next iteration you can adjust the number of tasks you undertake using an improved figure for your programming 'velocity'. This figure will be used to relate calendar time to the units of time used to estimate tasks and stories.
Planning the next iteration
In an XP project, you are never more than two or three weeks away from a major milestone, as the iterations are short and conclude with a release of the product, fully tested and production-ready. The current iteration is scheduled to complete at the end of the week with the release of a new beta test version. However, before then you need to help plan for the next iteration by estimating the time it will take you to complete some of the customer's new candidate stories. You also need to arrange these story cards so that the business can gauge the risk associated with each one by your confidence in the estimate.
The completion date and scope of an iteration is decided by the business. The story cards will be sorted in terms of those that are essential to the functioning of the product, those that add significant value to the business, and those that are 'nice to have'. Given that a functioning product must be delivered at the end of each iteration, the earliest completion date is the sum of estimates for stories in the first category divided by the team's 'velocity'-adjusted work capacity. The business must then decide what further stories are to be included in the release by balancing the cost to the business of delaying the availability of the product against the value of the functionality it will deliver.
You are anxious to produce better estimates for the next iteration, to avoid the situation that occurred a week ago, when you had to ask the business to reduce the scope of some stories so that the release deadline could be met. In XP, a deadline is never missed. Using quality or money as levers to control a project is not considered to be effective, and as time is fixed, this means that XP projects are steered by adjusting scope; this is done a little and often.
Working just a forty hour week
When you've finished planning, another developer asks you for help with one of their tasks, so you spend a couple more hours working as a programming pair to finish your day. XP does not permit anyone to work overtime for more than a week at a time and as you spent sixty hours last week trying to make good your schedule, this week you're limited to a standard forty hours. It's 4.30 in the afternoon, and your working day is done. It's not been a bad day; one of your tasks is complete, you've estimated the remaining candidate stories for the next cycle, you've helped someone else complete one of their tasks, and you'll still be home in time to have a workout in the gym!
Pros and Cons
XP works very well in a rapidly-changing world where requirements are vague or unknown. The business can adapt the product to meet its needs iteration by iteration, rather than producing a definitive specification up-front. It is not the sort of process you would use for building a safety-critical system, as it lacks the necessary traceability. However, if you want an agile process, targeted towards teams of between three and twelve people, producing systems with quantifiable business value, then XP must be worth considering.
The biggest problem with XP is the demands placed on the person with the customer role in the team. It expects someone who has been plucked out of a line role in the business to take much of the responsibility for delivering a system that gives value to that business. This person writes the stories and produces the functional tests to prove they have been completed, so any criticism about what the system does is directed towards them. Customers find themselves doing a completely different job in an alien environment, often without training or support, so it is not surprising that there are reports of their burn-out in an XP team.
To make XP work, you need to keep the cost of change low throughout the life of the project, otherwise later iterations take increasingly longer to deliver less. XP supporters assert that using new technologies and adopting sound practices can fundamentally change the nature of your software development, so allowing this objective to be met. Its detractors point to the problems that crop up when XP is scaled-up beyond a certain size of project as evidence that the cost of change does eventually rise, and then more steeply than in other processes.
The arguments as to whether or not XP is a good thing seem set to continue. However, if you're not enjoying following your current process (or don't even have one) then perhaps XP might help, for above all it is meant to put the fun back into developing software.
Will Stott is a Chartered Engineer who moved into software over twenty years ago after building a Nascom-1 and hand coding his first program in hex. Today, he mainly programs in C++ and C# and is a firm advocate of agile process. Find him at www.willstott.com.
The Twelve Core XP Practices
- The Planning Game - The business is responsible for deciding the date, scope and priorities of the release using information from the developers about the size, risk and progress of the work.
- Small Releases - Put a release into production as soon as possible and then incrementally improve it with regular new versions released every month or so.
- Metaphor - Capture the nature of the product in terms of a high level image that can be understood by everyone, and which sums up how things should work.
- Simple Design - Look for the simplest possible solution and defer complexity for as long as possible. KISS - keep it simple, stupid!
- Testing - Developers continuously structurally test their code, and customers regularly functionally test the features they have specified, so changes are always made to a working product.
- Refactoring - Banish duplication and whenever possible make the program simpler, more flexible, and easier to understand while ensuring it still passes all its tests.
- Pair Programming - Two programmers sitting at the same machine continuously peer review each other's work.
- Collective Ownership - All developers can change any part of the code base at any time to ensure everyone becomes familiar with all areas of the code.
- Continuous Integration - Integration issues are minimised by rebuilding the system many times a day, so problems are resolved when they occur rather accumulating at the end of the iteration.
- 40 hour week - The team is paced by scheduling for less than a standard working week and banning any overtime two weeks in a row.
- On-site customer - Promote communication between the business and the developers by having a customer working full time in the team.
- Coding Standards - Code is written according to agreed rules that encourage better communication amongst all the people who will eventually work with it.
Process, Process, Process
Rules define the way we make things; who does what, how tasks are performed, the steps that need to be taken, the order in which things are done, what is kept and what is thrown away. A process is a collection of such rules allowing some form of complex group activity to be managed as well as adapted and improved for different needs and environments. The idea of defining a process for the development of software products makes a great deal of sense, but successfully putting it into practice is not easy.
The Software Engineering Institute promotes the Capability Maturity Model (CMM) to help organisations define and improve their process. This model has five levels and over a number of years you are encouraged to progress from the initial level to the optimised level by first implementing basic management control, and then going on to define, manage and finally control your process. It is said that if you've never heard of the CMM then you're at level zero - chaos! There are other organisations that promote process, but they are often just concerned with certification, such as ISO 9001. XP is accepted as being a process that largely satisfies the key process areas defined by the lower three levels of CMM, and using it does not preclude an organisation from reaching the higher levels of process maturity.
People who have worked on large projects in big organisations are usually more familiar with the ideas of process, methodology and life-cycle. In a large project there is a need to structure development according to some accepted model or methodology. For example, the classic 'water-fall' life-cycle describes development in terms of sequential phases of analysis, design, coding and testing. The need to control the complexity that often arises during these phases led to the adoption of methods like structured analysis or object-oriented design. This combination of life-cycle and methods, together with a need for the tools to make them work together, created a market for standardised processes; something that could be bought off-the-shelf and adapted to meet an organisation's aspirations for defining their own software development process.
Standardised processes like the Rational Unified Process (RUP) and the Select Perspective are characterised by the creation of models to describe the system to the different people involved at different stages of the project life-cycle. For example, during analysis the business might create a Use Case model, which is then used by programmers to produce various UML models, before being turned into a set of skeletal class files for the coders during implementation. In a large team this modelling helps everyone to communicate as well as to understand and trace the underlying requirements. However, in a small team the endless modelling and bureaucracy of 'big process' can become an intolerable burden slowing progress to a crawl. In recent years the move towards smaller and more dynamic projects has created a demand for agile processes, such as XP, better suited to this type of environment.
Before you rush off to introduce a new process into your organisation, be aware that establishing a software process requires a significant investment over several years, often without any measurable return in the short term. Your old, tried-and-trusted methodology practised meticulously and faithfully will usually yield better results than any new process introduced in a rush and without sufficient commitment. There is no doubt that having a successful process will pay off in the long term, the question is whether your job will last that long!
History of XP
Kent Beck is the founder of XP, although in his seminal book on the topic, Extreme Programming Explained (1999), he claims most of the practices are as old as programming itself.
He credits Ward Cunningham, Ron Jeffries, Martin Fowler and Erich Gamma for providing him with much of the source information that led him to XP. The first major project to try XP was the development of a payroll application called C3 at Chrysler in 1996, when Kent Beck was parachuted into a team that had run into trouble. His success in turning the project around was attributed to XP, and this became the proof by example needed for it to gain widespread recognition.
Over a dozen books have now been written about XP, and it has become established as a firm leader of the agile process movement.
Sources of further information
- Kent Beck, Extreme Programming Explained, Addison-Wesley, 1999 (ISBN 0-201-61641-6). The original XP book is an essential read for practitioners.
- Mark Paulk, Extreme Programming from a CMM Perspective, Paper submitted for XP Universe Conference, July 2001. This paper reviews XP from the perspective of the influential SEI Capability Maturity Model
- Ward Cunningham's web site contains many XP references.
- Ron Jeffries' web site is also a prime source for information about XP.