Dealing With Trouble While Adopting XP
The XP process allows so much flexibility in getting things done but this comes at the cost of focus. I've previously talked about how product managers have to be careful they don't abuse the flexibility in XP, but recently Ephox has discovered the huge benefit that being focussed can be, even when you're doing XP.
We've come up against a very tight, very important deadline for a really big, really difficult new feature and at the same time we lost our entire support department so all support fell into the engineer's laps. Support is just one of those things you have to do as a software company and our first impression was that the extra support load would reduce our velocity but we'd be able to share the load around the team, get the day's cases out of the way and get back to developing. What actually happened was the entire engineering team wound up doing support all day and development stopped. Worse still, the team felt drained and tired all the time. We should have been able to get support out of the way faster and get developing, but for some reason we couldn't.
The problem was a lack of focus. Instead of coming in each day looking forward to getting our teeth stuck into the really big, difficult feature and making progress, we came in wondering how much time support would take up. On top of that is the impact of context switching - every time you start a new case you have to come up to speed with a new customer's use case, their problem, a new area of the product and often, another product. All that context switching not only takes time, it tires out your brain. By the time you get to actual development, you're brain has had enough for the day and just isn't capable of working on really big, really hard new features.
So what have we done about it? We split the team in half. Half the team spends all their day doing support so that we keep up our high levels of customer support. The other half spends all their day doing development so we can actually have a chance of getting that really big, really difficult feature done. We've only been working this way for a week now but we've made more progress in that week than we have in the prior month. Instead of feeling like the big feature was turning into a big death march, there's some hope of getting it out on time.
There's some very serious down sides to this approach though. We've just sacrificed half our team to doing support - that's a tough job to do, particularly when you're a passionate developer that loves creating new stuff. So it's a big ask for them to just focus on support and the team and the company needs to appreciate that - a lot (and we do). The other major downside is the lack of knowledge sharing. We had been rotating pairs around the entire team and we were making real progress at bringing everyone up to speed with all our products. We've had to sacrifice that to make our short term goal and it will come back to bite us in the future. Friday afternoons is now code review time where we go back through what we've done for the week and explain it to the rest of the team - it's not as good as pairing with them, but it's better than nothing.
The most serious risk with this though, is the risk that quality will drop. There's no point in shipping this really big, really difficult feature if it isn't also really good and really reliable. We need to keep up our discipline while under the stress of a tight deadline and while we don't have the whole team providing their unique skills to the development work. That means a lot of discipline and constantly checking that we're doing everything we can to get things right the first time.
We'll definitely be constantly reviewing the product quality, knowledge sharing debt we're accumulating and how the team is feeling to make sure we don't sacrifice too much long term potential for a short term gain.
August 15th, 2006 at 10:33 am
I spent the last year working in an solid Agile/XP environment. My conclusion? Amazingly enough, I don’t believe it is any better than any other methodology. The problem? It is still a methodology. It prescribes a specific set of things that may or may not always be appropriate.
The theory sounds great and it might even seem to work great over the first couple of sprints. However, we found ourselves hurt by an entirely new set of problems with Agile. For example, we were always overloaded with maintaining a complex, automated build, unit test, integration test, and acceptance test environment.
Some developers spent more time trying to do TDD than actually writing code. This wasn’t always a lack of experience, either. I’m a firm believer that some code doesn’t lend itself to TDD. I also don’t think a rule that mandates 100% test coverage is feasible.
Pair programming never resulted in a large improvement in productivity, even over the long term. We never got twice the productivity from a pair.
The products severely suffered from the lack of a QA period. A QA period is strictly forbidden, and as a result the system never got tested properly. It is ridiculous to say that TDD will result in perfect, bug free code.
The products often suffered from architecture problems. Since you only design a month’s worth of a system at a time, the “big picture” often got missed and you ended up with a patchwork system and no mechanism to go back and clean it up.
Senior developers were often overworked doing the work that managers usually do. When they say “the team self manages” that really means that senior guys have to step up and do a lot of “people management” since there is no manager to do it.
We had adopted the methodology 100% and had a lot of resources to help us including some big names from the industry. There are always answers given to all the things I mentioned above. Regardless, we had a talented, committed and qualified staff but in our retrospectives we always sat around talking about how it wasn’t working.
I have my own theories about how to develop software efficiently and effectively, but I’ll save that for later :)
August 17th, 2006 at 5:21 pm
[...] There's a lot more that has changed recently that has led to our sudden improvement in velocity - not just pair programming. For a start, we're more focussed on the task because we're not sharing the support load around the entire team. Plus, the business has really started to understand how difficult it will be for us to get things done so they've gotten off our backs about everything else and really focussed on hitting this big scary deadline. We were splitting our development efforts between two or three products and the context switching was killing us. [...]
August 28th, 2006 at 11:06 am
Brian Young says that using an Agile development practice such as XP, “A QA period is strictly forbidden, and as a result the system never got tested properly. It is ridiculous to say that TDD will result in perfect, bug free code.”
Brian, you’re quite right to say that it’s rediculous to expect TDD to result in perfect, bug fee code. But did you miss the recommendation for acceptance testing? In all of the agile approaches I know, the QA period covers the entire iteration, working in parallel with development. It’s too important to leave as a final phase, and is less effective when left as a final phase. Better to catch problems as early as possible, when they can be dealt with systematically instead of by brittle patches, and before more code is built on top of them. The “throw the completed design over the wall to QA” approach has never worked well, in any field.
August 30th, 2006 at 9:02 am
George, I know where you are coming from. I agree, a waterfall cycle is very ineffective when the QA team sees a beta release for the first time after 6 months of development. However, I dispute the agile belief that continous testing (unit, integration, acceptance) can completely eliminate the need for any “final” testing.
What do I mean by “final” testing? Take this scenario: the theme of a 30 day sprint is to produce a core component of a graphical display that a range saftey officer at a missle range will use to do his job during a launch. Stories are knocked out one by one until only a couple remain on day 29 of the sprint. Developer Joe is working late that night and runs into trouble. He has to change some base classes used by all the other stories in order to finish his story. He changes them, and the CI system runs all the sets of tests at midnight for the final build. Everything compiles and passes. Out pops the golden egg: an InstallShield script runs, and bam! a CD is cut.
After the sprint demo, the software gets installed at the missle range (it has been “tested”, right) and then used in production. Nobody ever takes the CD and installs it and does any further testing- after all, the next sprint has already started! Does this seem like a good idea? I don’t think it does but agile doesn’t seem to have an answer for this.
The fact that a new sprint starts as soon as one ends really led to burn out. There was never any “down time” to do any cleanup, maintenance, additional testing, etc. Agile says you do this stuff as you go along. In my experience this isn’t always the way it works out, and when it doesn’t you’re in trouble. You start utilizing hours upon hours after 5pm to take up the slack and eventually, people start quitting.
August 30th, 2006 at 10:23 am
Brian,
There is no reason you shouldn’t have a QA department in XP - they are the customer’s assistant and they can do QA after the end of the iteration. If you are deploying non-massively-critical internal systems you may not need this as the test suite and continuous testing is enough to make sure that the system doesn’t destroy day to day productivity, but any smaller bugs the slip through don’t matter too much. In safety critical areas like missile control, obviously you should be doing a ton of QA during and after development. Not doing so is foolhardy.
Regarding people getting burnt out because there’s no time between iterations, there are two concepts that are used in XP to avoid this. Firstly, sustainable pace says you shouldn’t be doing all that work after 5pm, you are fresher and more productive if you go home on time. If you find your team is burning out you are not working at a sustainable pace and you need to find ways to reduce stress and burn out - how you do that will depend on the particular team.
The other important concept is slack. Having slack in your schedule each iteration allows you to do the continual maintenance and additional testing etc that you saw happening after work hours. The key thing is to admit to yourselves and the rest of the business that these tasks exist, they take up time and they are very important so you have to leave time in the schedule for them to happen. If you find they aren’t happening, pull back the amount of work you take on for each iteration to give you more time to do them.
Agile methods depend on doing all of the practices together - implementing or focussing on each one as it becomes important. In other words, you may not roll out all the practices at once (it’s too much for the team to handle all at once) but as you see problems cropping up, roll out or improve the practice that solves that problem. In the case of your missile guidance you needed better customer acceptance tests, better unit testing and a better QA system backing up the customer. In the case of your burnt out team, you needed more slack and a focus on sustainable pace. Agile methods provide the building blocks to solve these problems, you just need to be sure you continuously review what you’re doing and adjust your processes to constantly make things better.
Basically, if after 6-12 months of doing XP you’re still doing it the way it says to in the book, you’re not doing XP because you’re not doing the review and improve part of XP that is so critical (mostly this comes through in retrospectives).