Symphonious

One of the challenging practices in XP is daily deployment - it requires your development team to have a very low defect rate and completely automated build and deployment tools. In an off the shelf software scenario it has the additional challenge that you can't actually get your customers to upgrade every day.

At Ephox we've adhered to this practice pretty well in the past by automatically deploying builds to all our internal systems, including our corporate wiki, website CMS and even this blog. Of course, that's still just a small subset of the kind of environments that we're actually used in and it's not actually getting out to real customers. To take the next logical step, we need to make those builds available to any customers who feel like checking out what's new - maybe not every day, but between all our customers hopefully regularly enough to give us good feedback.

So that's what we've done with the EditLive! Early Access program on LiveWorks! If everything is going well and the engineers are actually working on EditLive! (they do like to go home on weekends) a new build should be automatically deployed to LiveWorks! twice a day ready for people to try out. We also include the current change log on the page (again, automatically updated) but at this stage we're only listing changes from the last released version rather than every little change we make in the change log so it doesn't necessarily show changes in every build. We might need to look into publishing commit logs as well in the future to work around that.

Personally, I think this is pretty exciting - there's a lot of cool stuff that gets developed early in the release process that previously hasn't seen the light of day until months later. Now it's up on LiveWorks! within 12 hours.

As always, if you have any problems, suggestions random doodlings on the matter, comments are below, my contact details are just over there on the right and there's also the LiveWorks! mailing list.

So Doug is excited about how we took our first steps in a new product and how well it went. Personally, I'm impressed with the way that we presented all the usual engineering setup tasks to the client in a client focussed manner. We could have done it better by not discussing up front all the engineering tasks we were hiding behind the suggested first story, but that's okay. The first story was that we wanted to ship a distribution of the new product. It's really quite backwards to think of things that way - normally you build the product then work out how to package and distribute it, but it's impossible to ship anytime if you don't build the distribution mechanism at the start.

So hidden in that story was the following engineering tasks:

• Create a subversion repository.
• Create a build process (including managing build numbers, version numbers etc).
• Create a change log.
• Create zip file for distribution.
• Add the project to cruise control so it is automatically built.
• Adjust our "AutoDeployer" to be able to deploy the built zip file to the web site.

We also found time to pay back a bunch of technical debt left around our systems:

• Cleaned up some cruft in our Subversion repository that we don't use anymore.
• Fixed the build process for our IWWCM integration so it runs about 10 times faster and builds only trigger when appropriate - they were triggering on unrelated changes in subversion.
• Cleaned up and improved the AutoDeployer.

Not to mention, we've learnt a lot about starting new products from scratch and gotten that process up and running quite well. I guess you could say I'm excited too.

I’ve seen a misconception a number of times where people believe that the regular release cycles and TDD practices in XP mean that you don’t need a standard QA process - that the developers are responsible for writing perfect code, first time every time. It’s true that XP practices can significantly improve quality, both of the code and the final product, but that doesn’t excuse the team from properly testing their work.

It’s important to note that TDD is a design process, not a testing process. It will only test the cases that the initial developer thought of. For changing code or when the developer can’t keep all the relevant cases in their head, the tests will eliminate regressions, but by definition they won’t cover any cases the developer didn’t expect. Somewhere along the line you need someone who’s not as close to the code to really test the code and cover all the cases - not just the ones the developer thought of. Pairing helps with this, but the pair is still too close to the code. To do it properly and ensure you’re releasing a quality product, you need an actual QA process.

Regular releases in XP make traditional QA processes impossible to work with - you can’t finish all the features, throw them over the wall to QA, fix all the bugs and repeat inside a one week iteration. Instead, QA has to be an ongoing process where the QA department works alongside the development team. Straight after the planning game, QA should assist the client in writing acceptance tests that provide in-depth testing of the story requirements to ensure all the corner cases are covered. Clients often don’t think of corner cases off the top of their heads so having someone with decent QA skills will make the acceptance tests much more detailed and ensure that when the story is signed off it actually does what was needed.

As the developers work on stories, QA needs to keep an eye out for any clarifications to requirements and extra cases that the user explains to the developers while they work and make sure they are reflected in acceptance tests.

Finally, once a story is completed by the developers, QA should become an advocate for the client and test out the feature for anything that looks wrong while the client goes through and checks that it meets their requirements. This is where anything that can’t be covered by an automated test gets tested and the testing requirements documented so those tests can be run manually regularly.

During their slack time, the QA department should research ways to automate the currently manual tests, go back and run the manual tests to make sure they still pass.

The QA department might be a group of specialists or they might be the same people as in the development team - it’s the process and the allocation of time that matters more than the people. However, QA requires a distinct set of skills to development and a lot of developers aren’t much good at doing QA. If your development team doesn’t have QA skills you need to either train them up or get in QA specialists. My general inclination is that it’s best to have QA specialists who have a real passion and talent for testing software, but that your regular development team should regularly pair with them on QA work (and them with developers on non-QA work) to make sure that the team values and understands the work they do. It’s also important to avoid a divide between developers and QA - both teams are there to deliver value to the client, not to fight about whether a behavior is a bug or a feature request etc. QA and development need to be one team that works together and sits together.

One other advantage of keeping QA and development working together, is that developers fix the bugs that QA finds straight away instead of building up a big backlog that then delays the release while all those bugs are fixed and verified. Keep on top of your bug list and QA processes and you can ship top quality software in predictable timelines without the death march of bug fixing that traditionally happens at the end of projects.

One of the really core principles of XP is the idea of small, frequent releases. The most obvious benefit of this is that it allows users to see the product and provide feedback about what could be done to improve it. What's not so obvious, but is part of the reasoning behind a number of XP principles, is that the developers benefit from this just as much, if not more than the users. Being couped up in an office somewhere slaving away creating software that no one's using just isn't fun. Getting your software out to users and having the feeling that you've achieved something is a big feel-good moment and it energizes the development team, allowing them to keep going full pace instead of getting tired and stressed.

This feeling of accomplishment doesn't just come from an actual release though - even completing a story makes you feel like you've added value. The last few weeks I've experienced first hand how deflated you get without those little wins. I've been focussed on a big complex area that was too experimental and unknown to break down into small user stories with any real success. It really felt like a death-march - I had no idea how much further I had to go before I finished and as each day passed I felt more and more like I wasn't making any progress at all. Today I suddenly reached the end, the feature was done. Although it's overly dramatic, it reminds me of a single passage about stepping onto the summit of Everest for the first time: after weeks of looking up at the slope ahead of them, for the first time they were looking down. I'm going to have to find a way to avoid such long hauls in the future.

According to XP, what I should have done at the start was a spike to find out what the major obstacles were and find a way to break the story down into something manageable. We did in fact have the story broken down into smaller sections, but it didn't reflect the actual work and I wound up with a pile of six or seven cards that I felt I had to work on simultaneously. Perhaps I could have been more focussed on a single story and just done what was required for it, but looking back at what was required, I don't see that being a much more successful approach. The simplest approach that would have worked for some of the stories would have had to be rewritten to work for the later stories. I also don't know how I could have approached a spike to identify the problems that I ran into, most of them came completely out of the blue.

In the end, I really don't know how I'd do it differently next time, I just know that I want to do it differently because it made me feel bad. It may turn out to be the quickest way to get that particular feature done, but the fact that it made me feel bad feels like a warning sign that it's not a sustainable pace. Perhaps next time we'll be in a better position and be able to share the work around the team more so it doesn't feel like a burden on one person's shoulders. I'd certainly feel better with that if only because it didn't make me feel like a prima donna - I've been so focussed on this one task that I've been delegating pretty much everything else to other people. Fortunately, my colleagues have been fantastic at doing what needed to be done. Thanks guys.

Slashdot linked off to an old interview with Kent Beck and Cynthia Andres, but it was interesting to see the number of people complaining about the pair programming aspect of XP. Comments like:

Programmers are solo beasts - putting two of these dragons behind one keyboard is asking for trouble.

and:

In the OO Programming course I had this year, they encouraged us to practice extreme programming. Well, it sucked… Everytime my partner and I encountered a little problem and one of us had an idea, he had to explain it to the other, which took at least 3 times as long as just typing it out, which in almost every case made the idea perfectly clear. Suffice to say, after about 4 hours we gave up and just sent each other any files we had edited (of course making sure we didn't work on the same file at the same time).

show a major lack of understanding, not only of pair programming, but of the real world in general. Firstly, there's no reason when pair programming that you shouldn't just type out your idea instead of spending excessive amounts of time waving your hands around and trying to explain it. Code talks and tests talk even better, so use them as a method of explaining. Secondly, and most importantly, if members of your team can't effectively work together and communicate clearly, you have a major problem with your team that will reduce your chances of success in any methodology. In XP such a scenario almost eliminates any chance of success.

It's not just your chances of success that drop when you have a poorly gelled team, it's also your teams morale and work satisfaction. XP should be fun to do. If it's not, you either aren't doing it right or you have a bad team. At Ephox, we tend to spend a lot of our day laughing and generally being happy. That atmosphere comes both from having a gelled team, but also from a number of XP practices that encourage social interaction.

The most significant practice in this regard is sitting together. Being locked away in your own office means that you have to make a special effort to interact with your work mates. Not only do you lose the serendipitous communication of overhearing important stuff, but you also lose the social interaction, the personal touches, that turn work mates into friends. Doesn't that mean that you waste time getting interrupted by talking about what happened on the weekend? Sure does. Does it mean you enjoy yourself at work more, have better developer retention rates and ultimately go faster because happy workers are productive workers? You bet.

Your pointy haired boss might want you to have your nose to the grindstone constantly, but that's because it's notoriously difficult to measure actual productivity so instead pointy haired bosses use number of hours on task as the measure. If they can make you work longer hours and never be distracted from your task then obviously you'll be more productive right? For some things like mind-numbing production line work that is probably true. For work where you need to your brain to be sharp and fresh, not a chance. When you need to think clearly, you need to give you brain a rest regularly and not push yourself to hard. When you do work, it should be energized work.

If you want to be successful with pair programming, you need to get on well with your work mates. You don't have to be the best of friends, but you need to be able to sit and chat, laugh about things and be comfortable around each other. Pairing with a team full of prima donnas and antisocial types would be a nightmare1 so it's vital that you focus on building a gelled team, not just a team who can solve silly math puzzles.

You might think that this is a requirement just of XP, but it's not. It's a requirement of enjoying your work. Even pointy haired bosses know that enjoying your work is important, why do you think they spend so much on company christmas parties and the like? What they don't realize is that one-off events are an insignificant impact on morale compared with having teams that enjoy working together all the time. When done well, those one-off events are designed to get the team to come together, get to know each other and gel better.

At Ephox we consider a gelled team so much a necessity that we make the last step in our interview process going out with the entire engineering team for coffee. There's no technical questions, just sitting around chatting with the people you'll work with. Earlier parts of the interview process will tell us if you have the right skills, but they only give us a vague idea of whether or not you'll fit into the team2. Going out with the whole team though, lets us make sure that there's no major personality clashes with anyone in the team, and that you fit in with the team culture, spirit and people.

So what do you do if you're in a bad team? Either get rid of the antisocial types, or get out yourself and find a team that knows how to work together and have fun. Feel free to send me a résumé.

1 - actually, spending any time with prima donnas and antisocial types is generally unpleasant↩

2 - as a hint, your interview at Ephox has gone well if you and the interviewers have laughed through most of it. As a corollary, if you spend most of the interview laughing and the interviewers don't, you probably didn't do so well. It's bad in any interview if the interviewers spend the whole time laughing and you don't - but we're too professional to do that. ↩