Symphonious

There's a lot of talk about how you convert a legacy project over to TDD, and there's a lot of talk about how you TDD on little code examples, but how do you get started on a real project? For a swing app, how do you TDD public static void main(String[] args)?

Atomic testing wants you to be able to mock out your dependencies, which leads to things like dependency injection. Somewhere along the line though, someone has to create those objects and pass them in. How do you atomically test that?

If I decide to use struts (or rails or whatever) as the base framework for a web-app, what tests should I have to ensure that struts is working as I think it should and to ensure that I know if something I've depended on changed in a future release?

I understand a whole lot about TDD and I'm quite comfortable sitting down with an existing code base and adding features via TDD. Despite that, every time I think about starting a new project and doing it right from the start, I just feel completely lost. I have no idea how I go about starting. Are there any good guides on starting a TDD code base? Are there any code bases out there that actual test everything that could possibly break or do they all have that one nasty bit of code that sets everything up and that we just can't test?

In James Shore's chapter on Continuous Integration for his upcoming book, he strongly recommends using synchronous integration, where each time you check in you stop and wait for two builds to complete before moving on, in favor of asynchronous integration, where you check in and move on to your next task leaving an automated build server to check everything's still okay. The whole point of continuous integration is that your team integrates regularly, so that you avoid conflicts. I can't see how this is encouraged by adding up to twenty minutes of work1 every time you want to check in.

The advantage of synchronous integration is that you always know the build is broken - sort of. In reality you may find that the build broke, but you lock out all the other developers from accessing the source code repository while you check in and do the test build, about ten minutes every time someone checks in5. This lock out period provides the illusion that the build is always passing, because anytime someone checks out the code base or does an update, they get fully working code with all the tests passing. It's really useful to know that anytime you update that you won't get any unexpected surprises, but adding in this delay means less frequent integration cycles, more time waiting around and a greater chance of integration problems.

With asynchronous integration you accept the risk that every now and then the build might break and in exchange encourage people to integrate more often, allow people to integrate part way through a story without interrupting their flow2 and let the whole team know when a problem has come up. Letting the whole team know is an unexpected benefit, it means that when the build breaks, the whole team is informed that an unintended side-effect has been encountered and that help might be needed to resolve it. Most of the build failures we see are complex issues where a new feature is impacting on the expected results of an existing feature, and since we're working with a legacy code base the failing test may not make it obvious why the failure is important. People new to the code base may well have thought that the failing test just needed to be updated because what it was asserting was outside of its intended scope. Fortunately, since the whole team sees the problem, the more experienced people may recall why that was important and be able to provide advice on how to make the test clearer in its intentions and how to resolve the conflict in functionality.

Being able to check in more regularly is a big advantage as well. If the check in process takes twenty minutes to complete, you might check in two maybe three times a day, with the check in process taking one to two minutes, we check in five to ten times a day. That's a lot more regular integration and a lot less chance for conflicts. If the team is working on completely separate areas of the product that's probably not so useful, but often working on commercial products you find the features you're adding are related. Even with a well designed code base with limited coupling3, there's a good chance of conflicts when you've got the whole team working on improving the functionality of list handling.

There are a couple of major problem that many teams encounter with asynchronous integration, and James Shore points them out nicely. Firstly:

If the build succeeds, asynchronous integration does save time. However, if the build fails, you have to interrupt your new task to roll back and fix the old one. To do so, you must leave your new task half-done, switching context (and sometimes pairs as well) to fix the problem, then switching back. It's wasteful and annoying.

This is annoying when your check in occurs at the end of a story, if you check in regularly during the development of a story, when the build fails you're probably still working in the same area of code, with the same pair and can quickly identify why the build failed and fix it straight away. As a bonus, you've just learnt something about the area you're working in which is clearly important. The other way to mitigate this problem is to make the build fail less. That seems silly, but there are some simple steps you can take to make build failures far less common:

1. Don't program by coincidence. You may have tests as a net to catch you if you slip up, but that doesn't mean you should just randomly change code until the tests pass. You need to understand the changes you are making to the code base and have confidence that you haven't broken anything even without running the tests. Use the tests as a safety net, not as a replacement for intelligence. You will still make changes that cause tests to fail, but you will do it less if you think about what you're doing.
2. Identify the tests for the area you're working in and run them very regularly while you're developing. Keeping a custom test suite around is a good way of doing this, call it something like sanity-check and don't check it in. It's just a scratch pad that you add related tests to when you come across them. Also run these tests before you check in.
3. Identify the really fast tests and run them before you check in. What tests you use here depends on the state of your project, how you like to work and many other things. The key thing is that the tests run fast enough that you're not inclined to walk away from your computer while they run. Checking your e-mail or taking a quick mental break is good, but breaking flow is bad. They should run in under a minute. Some ways of picking these tests are:
   • All the atomic tests. If you can run all your atomic tests in under a minute, run them.
   • All the 'fragile' tests. It sounds bad to have fragile tests, but there are likely to be parts of your product that are more likely to break than others. Run these before you check in. Also consider the areas that these tests cover as potentially smelly and see if you can make them more robust.
   • The tests for the section of code you're working in. This is similar to the set of tests you build up in section 2, but will probably contain more tests in coarser grained sections. If you know that your menu bar is a completely separate module, you can make the menu bar tests separate too and run them before check in whenever you change the menu bar. Since it's completely separate, you should never see a build fail because of changes to the menu bar code if you ran all the menu bar tests.
4. Don't forget to add files. You might think that everyone forgets to add a new file every so often, or misses committing an important change - it's just a fact of life. That's the wrong way to think. Don't commit if there are any files that are unrecognized by your version control system - that means any files that are never checked in4 need to be added to the list of files to ignore. Then make sure you use tools that make it obvious that there are unrecognized files, or better yet just refuse to commit if there are. Finally, when you commit, always do so from the top level of the source tree and always commit every change. If you have changes that you don't want to commit yet, store a patch and revert them, then run the pre-check-in tests again. Don't have unrelated changes in your checked out code - either work on one thing at a time, or check out a new copy of the code for each simultaneous thing you're trying to do.

In practice, rather than switch gears in the middle of a task, many teams simply let the build remain broken for a few hours while they finish their new task. If other people integrate during this time, the existing failures hide any new failures in their integration. Problems compound, leading to a vicious cycle of painful integrations, leading to longer broken builds, leading to more integration problems, leading to more painful integrations. I've seen teams that practice asynchronous integration leave the build broken for days at a time.

Don't do this. Every build failure is critical and needs to be addressed immediately. If your team doesn't treat builds this way, don't use asynchronous integration. That doesn't mean that your team needs to panic when the build breaks, but it does mean that as a team, you have to make sure someone is immediately working on fixing it. The fact is, synchronous integration only works because the team wants the build to keep passing - otherwise people would just ignore the check-in process. Asynchronous works on the same basis, but requires an even higher level of commitment to the process. You can't use asynchronous integration to sneak continuous integration into your team, you can't dictate to the team that continuous integration will be done, you have to foster the commitment in the team before you implement the integration process. Make it a point of professional pride that you don't leave the build failing for long periods of time. Make sure there aren't time pressures that encourage leaving the build broken - make sure your client understands that fixing the build is always more important than working on their stories. If you can't do that, do synchronous integration and identify things that your developers can do to be productive while waiting on the build to complete.

Remember, three check-ins a day with a twenty minute delay on each check-in means an hour of downtime per day per developer. While you don't want to hold your developers to the grindstone constantly, that's a lot of down time, so you want to make sure they can do something useful while the build happens. They might check up on other pairs to see if they need help, handle their e-mail, read technical publications or catch up with RSS feeds, brainstorm new stories or products with the customer or discuss how they'll approach their next task. Some of the time they should probably take a short walk outside while the build happens, or go make coffee etc. Most likely though, they will need to do these things while working on a story when it gets tough and they need a break though, so don't assume all of the downtime will happen while the build is happening.

One important note is that whether you do synchronous or asynchronous integration, you need a fast build - if your build is too long with asynchronous integration you are more likely to get multiple changes bundled into the one build which makes it harder to identify the cause of problems. Keep working to make your build as fast as possible and monitoring what impact the build time is having on your team.

There's no one right way to do continuous integration with your team, you need to evaluate the process carefully and find what's best for you. If you start with asynchronous integration and find that the build stays broken for long periods then you should probably switch to synchronous integration. If you start with synchronous and find that you almost never have to roll-back a commit, you may be able to reduce the overhead of your integration process by switching to asynchronous integration. You may find that tests often fail but they fail consistently on the local machine and the build machine - you may need to run a full build locally but then leave the build on the integration machine to an automated system. Keep watching the health of your build and adjust your processes as needed to fit your teams needs.

1 - or more frustratingly, twenty minutes of waiting around↩

2 - obviously at a stable point where all the tests pass↩

3 - and let's face it, those are rare particularly with legacy code bases↩

4 - because they were created as part of the building or testing process↩

5 - ten minutes because the first build on your local machine is done before you take the integration token↩

I just stumbled across something interesting that I probably should have realized before. With TDD, the first thing you should do when you get assigned a bug to fix is write a test that reproduces it. This morning I was good and did just that, but surprisingly the test passed.

Now normally when your test passes straight off, it means you didn't write it correctly and you need to fix the test. In this case though I could confirm manually that the bug simply didn't happen - a classic works for me. Drop back into bugzilla and mark the bug WFM and I suddenly realized I could prove that it works correctly by pointing to the test.

So now, we have a closed bug and a passing test that will make sure it never comes back. The normal inclination I have is that the test passed straight off so it didn't have value and shouldn't be committed, but that's not true. Writing a new test for an area we didn't have a clear acceptance test1 for before is a really good thing and should be preserved. Better yet, I can point to that test in the bugzilla case and if someone disagrees they can change the test to show the problem and reopen the bug.

1 - I'd written the acceptance test first to verify the problem but didn't have a need to drill down to integration and atomic tests.↩

One of the most common things touted about XP is that it allows for rapid change. The client can change the requirements on the fly for a very low cost - they can add features, changes features and drop features regularly as the product is developed without incurring a massive cost of change. When you work on in-house applications or custom built software, that provides a really big benefit, but when you develop off the shelf software that means that the final release may contain a mish-mash of new features and be generally unmarketable.

So how do you take advantage of the ability to change that XP provides while still maintaining focus in your product development? Essentially it comes down to having a client, usually the product manager, that keeps track of the long term view and makes sure that while a few little features can be added, and things are improved to be the most useful for end users, on the whole the release stays on track to deliver a cohesive set of new features that user's will actually appreciate.

To a large extent, this means that the client, with the assistance of the rest of the business, engineering team and support team, plans out the aims for the release on a high level, and probably drilling down to some specifics before development starts. It's important to write down those aims so that you can bring them with you to each planning game to make sure that you're on track.

In XP terms, those goals are the yardstick by which you measure value. Each story needs to be evaluated against those goals to see how much value it brings to the release as a whole. The same approach comes out when doing in-house development since you tend to switch over sections of your processes to the new product at a time, so you are inclined to get each section fairly right instead of having a wide spread of largely useless features.

The flexibility of XP will allow you to suddenly add a key goal for a release because of competitive pressures, or to slip in that particular feature to close a big deal, but with some up front planning and an attentive product manager, you can still keep focussed on delivering a solid release.

In my previous post on tracking estimate accuracy, Stephen Thorne commented:

I thought the entire idea about estimates is that they should be self correcting? Estimates made in some arbitary numbering system, you sum up the estimates, divide by the time actually taken, then you have an accurate picture of how one developers estimates map to reality…

The ability to map arbitrary estimates to actual time that Stephen refers to is velocity. Essentially, how many points per time period can the team complete? This is a really effective technique if your estimates are consistent. That is, you may find that 3 points takes on average, a day to complete. So your velocity is 3 points per day1. However, if your estimates aren't accurate, you may have some 3 point stories take 15 minutes, and some take a week.

Clearly, estimates that vary by that much make scheduling and planning extremely difficult and reduce the confidence you have in your measure of velocity - what if one week you happen to pick 5 week-long 3s? To avoid this, it's important to try to make your estimates consistent. It doesn't really matter if a 3 is half a day or two days, as long as it is a consistent measure of work.

One of the most common ways that this happens is that the original estimate didn't take into account some prerequisite work that needed to be done. Addressing this problem is why we plan to start brainstorming tasks required when we estimate - hopefully by spending some more time thinking about what's required, we'll find less unexpected surprises.

Another common way that estimates turn out to be wrong is that the team has to pay back code debt as part of the story. This is significant refactoring that will pay itself back over time, but makes the task at hand involve more work than expected.

A less common, but particularly painful case is when unexpected or unplanned dependencies occur between stories. In an ideal world, all stories are independent of each other, however sometimes it doesn't work out that way and it turns out that another story needs to be completed before this one can be. This case doesn't really make for a bad estimate, as much as it throws out the planning for what can be completed. The estimates for each of the stories are probably right, but in the planning game, both stories should have been played in the same iteration instead of just one. As a result, the team needs to drop stories unless they get lucky and have a higher than expected velocity.

1 - Typically, velocity is tracked per iteration and is reported as just a single number rather than "per" some timeframe.↩