Both approaches require more floor space which makes them difficult to setup but I would expect the face to face pairing to be a heck of a lot better if done well. I’ve always preferred having separate screens in mirror configuration as well as separate keyboards and mice to allow the developers to sit a little further apart to be comfortable and to be able to look straight ahead at the screen. That said, I quite like having a second monitor for spreading out windows as we have at LMAX so it’s not clear cut which is better.

It’s also interesting to note the popularity of the flat screen iMacs as opposed to Mac Pros or laptops. The former being too expensive for extra power and extensibility that generally isn’t required and the latter often being a bit too individualised to be good as pairing machines. Plus laptops, while amazingly powerful these days still have less bang for the buck and the reduction in performance is just enough to matter for development.

At Ephox we've adhered to this practice pretty well in the past by automatically deploying builds to all our internal systems, including our corporate wiki, website CMS and even this blog. Of course, that's still just a small subset of the kind of environments that we're actually used in and it's not actually getting out to real customers. To take the next logical step, we need to make those builds available to any customers who feel like checking out what's new – maybe not every day, but between all our customers hopefully regularly enough to give us good feedback.

So that's what we've done with the EditLive! Early Access program on LiveWorks! If everything is going well and the engineers are actually working on EditLive! (they do like to go home on weekends) a new build should be automatically deployed to LiveWorks! twice a day ready for people to try out. We also include the current change log on the page (again, automatically updated) but at this stage we're only listing changes from the last released version rather than every little change we make in the change log so it doesn't necessarily show changes in every build. We might need to look into publishing commit logs as well in the future to work around that.

Personally, I think this is pretty exciting – there's a lot of cool stuff that gets developed early in the release process that previously hasn't seen the light of day until months later. Now it's up on LiveWorks! within 12 hours.

As always, if you have any problems, suggestions random doodlings on the matter, comments are below, my contact details are just over there on the right and there's also the LiveWorks! mailing list.

So hidden in that story was the following engineering tasks:

• Create a subversion repository.
• Create a build process (including managing build numbers, version numbers etc).
• Create a change log.
• Create zip file for distribution.
• Add the project to cruise control so it is automatically built.
• Adjust our "AutoDeployer" to be able to deploy the built zip file to the web site.

We also found time to pay back a bunch of technical debt left around our systems:

• Cleaned up some cruft in our Subversion repository that we don't use anymore.
• Fixed the build process for our IWWCM integration so it runs about 10 times faster and builds only trigger when appropriate – they were triggering on unrelated changes in subversion.
• Cleaned up and improved the AutoDeployer.

Not to mention, we've learnt a lot about starting new products from scratch and gotten that process up and running quite well. I guess you could say I'm excited too.

It’s important to note that TDD is a design process, not a testing process. It will only test the cases that the initial developer thought of. For changing code or when the developer can’t keep all the relevant cases in their head, the tests will eliminate regressions, but by definition they won’t cover any cases the developer didn’t expect. Somewhere along the line you need someone who’s not as close to the code to really test the code and cover all the cases – not just the ones the developer thought of. Pairing helps with this, but the pair is still too close to the code. To do it properly and ensure you’re releasing a quality product, you need an actual QA process.

Regular releases in XP make traditional QA processes impossible to work with – you can’t finish all the features, throw them over the wall to QA, fix all the bugs and repeat inside a one week iteration. Instead, QA has to be an ongoing process where the QA department works alongside the development team. Straight after the planning game, QA should assist the client in writing acceptance tests that provide in-depth testing of the story requirements to ensure all the corner cases are covered. Clients often don’t think of corner cases off the top of their heads so having someone with decent QA skills will make the acceptance tests much more detailed and ensure that when the story is signed off it actually does what was needed.

As the developers work on stories, QA needs to keep an eye out for any clarifications to requirements and extra cases that the user explains to the developers while they work and make sure they are reflected in acceptance tests.

Finally, once a story is completed by the developers, QA should become an advocate for the client and test out the feature for anything that looks wrong while the client goes through and checks that it meets their requirements. This is where anything that can’t be covered by an automated test gets tested and the testing requirements documented so those tests can be run manually regularly.

During their slack time, the QA department should research ways to automate the currently manual tests, go back and run the manual tests to make sure they still pass.

The QA department might be a group of specialists or they might be the same people as in the development team – it’s the process and the allocation of time that matters more than the people. However, QA requires a distinct set of skills to development and a lot of developers aren’t much good at doing QA. If your development team doesn’t have QA skills you need to either train them up or get in QA specialists. My general inclination is that it’s best to have QA specialists who have a real passion and talent for testing software, but that your regular development team should regularly pair with them on QA work (and them with developers on non-QA work) to make sure that the team values and understands the work they do. It’s also important to avoid a divide between developers and QA – both teams are there to deliver value to the client, not to fight about whether a behavior is a bug or a feature request etc. QA and development need to be one team that works together and sits together.

One other advantage of keeping QA and development working together, is that developers fix the bugs that QA finds straight away instead of building up a big backlog that then delays the release while all those bugs are fixed and verified. Keep on top of your bug list and QA processes and you can ship top quality software in predictable timelines without the death march of bug fixing that traditionally happens at the end of projects.

This feeling of accomplishment doesn't just come from an actual release though – even completing a story makes you feel like you've added value. The last few weeks I've experienced first hand how deflated you get without those little wins. I've been focussed on a big complex area that was too experimental and unknown to break down into small user stories with any real success. It really felt like a death-march – I had no idea how much further I had to go before I finished and as each day passed I felt more and more like I wasn't making any progress at all. Today I suddenly reached the end, the feature was done. Although it's overly dramatic, it reminds me of a single passage about stepping onto the summit of Everest for the first time: after weeks of looking up at the slope ahead of them, for the first time they were looking down. I'm going to have to find a way to avoid such long hauls in the future.

According to XP, what I should have done at the start was a spike to find out what the major obstacles were and find a way to break the story down into something manageable. We did in fact have the story broken down into smaller sections, but it didn't reflect the actual work and I wound up with a pile of six or seven cards that I felt I had to work on simultaneously. Perhaps I could have been more focussed on a single story and just done what was required for it, but looking back at what was required, I don't see that being a much more successful approach. The simplest approach that would have worked for some of the stories would have had to be rewritten to work for the later stories. I also don't know how I could have approached a spike to identify the problems that I ran into, most of them came completely out of the blue.

In the end, I really don't know how I'd do it differently next time, I just know that I want to do it differently because it made me feel bad. It may turn out to be the quickest way to get that particular feature done, but the fact that it made me feel bad feels like a warning sign that it's not a sustainable pace. Perhaps next time we'll be in a better position and be able to share the work around the team more so it doesn't feel like a burden on one person's shoulders. I'd certainly feel better with that if only because it didn't make me feel like a prima donna – I've been so focussed on this one task that I've been delegating pretty much everything else to other people. Fortunately, my colleagues have been fantastic at doing what needed to be done. Thanks guys.

Programmers are solo beasts – putting two of these dragons behind one keyboard is asking for trouble.

and:

In the OO Programming course I had this year, they encouraged us to practice extreme programming. Well, it sucked… Everytime my partner and I encountered a little problem and one of us had an idea, he had to explain it to the other, which took at least 3 times as long as just typing it out, which in almost every case made the idea perfectly clear. Suffice to say, after about 4 hours we gave up and just sent each other any files we had edited (of course making sure we didn't work on the same file at the same time).

show a major lack of understanding, not only of pair programming, but of the real world in general. Firstly, there's no reason when pair programming that you shouldn't just type out your idea instead of spending excessive amounts of time waving your hands around and trying to explain it. Code talks and tests talk even better, so use them as a method of explaining. Secondly, and most importantly, if members of your team can't effectively work together and communicate clearly, you have a major problem with your team that will reduce your chances of success in any methodology. In XP such a scenario almost eliminates any chance of success.

It's not just your chances of success that drop when you have a poorly gelled team, it's also your teams morale and work satisfaction. XP should be fun to do. If it's not, you either aren't doing it right or you have a bad team. At Ephox, we tend to spend a lot of our day laughing and generally being happy. That atmosphere comes both from having a gelled team, but also from a number of XP practices that encourage social interaction.

The most significant practice in this regard is sitting together. Being locked away in your own office means that you have to make a special effort to interact with your work mates. Not only do you lose the serendipitous communication of overhearing important stuff, but you also lose the social interaction, the personal touches, that turn work mates into friends. Doesn't that mean that you waste time getting interrupted by talking about what happened on the weekend? Sure does. Does it mean you enjoy yourself at work more, have better developer retention rates and ultimately go faster because happy workers are productive workers? You bet.

Your pointy haired boss might want you to have your nose to the grindstone constantly, but that's because it's notoriously difficult to measure actual productivity so instead pointy haired bosses use number of hours on task as the measure. If they can make you work longer hours and never be distracted from your task then obviously you'll be more productive right? For some things like mind-numbing production line work that is probably true. For work where you need to your brain to be sharp and fresh, not a chance. When you need to think clearly, you need to give you brain a rest regularly and not push yourself to hard. When you do work, it should be energized work.

If you want to be successful with pair programming, you need to get on well with your work mates. You don't have to be the best of friends, but you need to be able to sit and chat, laugh about things and be comfortable around each other. Pairing with a team full of prima donnas and antisocial types would be a nightmare1 so it's vital that you focus on building a gelled team, not just a team who can solve silly math puzzles.

You might think that this is a requirement just of XP, but it's not. It's a requirement of enjoying your work. Even pointy haired bosses know that enjoying your work is important, why do you think they spend so much on company christmas parties and the like? What they don't realize is that one-off events are an insignificant impact on morale compared with having teams that enjoy working together all the time. When done well, those one-off events are designed to get the team to come together, get to know each other and gel better.

At Ephox we consider a gelled team so much a necessity that we make the last step in our interview process going out with the entire engineering team for coffee. There's no technical questions, just sitting around chatting with the people you'll work with. Earlier parts of the interview process will tell us if you have the right skills, but they only give us a vague idea of whether or not you'll fit into the team2. Going out with the whole team though, lets us make sure that there's no major personality clashes with anyone in the team, and that you fit in with the team culture, spirit and people.

So what do you do if you're in a bad team? Either get rid of the antisocial types, or get out yourself and find a team that knows how to work together and have fun. Feel free to send me a résumé.

1 – actually, spending any time with prima donnas and antisocial types is generally unpleasant↩

2 – as a hint, your interview at Ephox has gone well if you and the interviewers have laughed through most of it. As a corollary, if you spend most of the interview laughing and the interviewers don't, you probably didn't do so well. It's bad in any interview if the interviewers spend the whole time laughing and you don't – but we're too professional to do that. ↩

At some point though, you get an error about the max permspace being reached. You've created so many mock classes that the JVM no longer has room to load class definitions. What's worse, your atomic tests which should be running lightning fast because they're so atomic, aren't. This is the point where you learn that Mocks are in fact a sometimes food. They're a nasty hack which allows you to pretend the world is all happy and isolated even though it's not.

Now, there are certainly architectures you could use which would completely avoid the need for mocks while still allowing inter-class communication but it can become mind-bendingly difficult to work out where the heck those messages are going and the performance implications of breaking things up at this level is pretty severe. A completely separated design is often beneficial at a component level, but it's usually impractical at the class level.

So mocks are a useful tool to help test things that need to interface with other classes but are largely independent. I've taken to the rule of thumb that if I need to use more than one mock in a test I'm probably doing something wrong – if I'm using more than three mocks I really need to rethink my approach.

How do other people approach the use of mocks and other related techniques?

1 – I'm not yet sure, but perhaps it would be better to focus on making them run as fast as possible, rather than as atomic as possible. They would still need to be focussed enough to make it clear where the problem is instead of just showing that this is a problem.↩

2 – possibly hidden away in the language runtime↩

Note: When you're TDDing and getting the very loose coupling every one is longing for ;-) you must be aware that integration tests and acceptance tests are an absolute necessity. Since this is the only way to really test your configuration.

On the other hand, to be able to get rapid feedback you want a fast build – under 10 minutes. From James Shore's draft of the 10 Minute Build chapter of his upcoming book, The Art of Agile:

For most teams, their tests are the source of a slow build. Usually it's because their tests aren't focused enough. Look for common problems: are you writing end-to-end tests when you should be writing unit tests and integration tests? Do you unit tests talk to a database, network, or file system?

You should be able to run about 100 unit tests per second (test_driven_development). Unit tests should comprise the majority of your tests. A fraction (less than 10%) should be integration tests, checking that two components synchronize properly. Only a handful, if any at all, should be end-to-end tests (testing).

The problem is, if you want to test your software comprehensively and be able to have confidence that your tests will tell you if you've broken something, I can't see how you can avoid writing a lot of integration tests. I also don't see why you would avoid automating end to end tests and running them very regularly. The reality is that you need to have a QA process that tests the application from how users actually use it – not from the point of view of this bit of the system or that bit of the system. You need to verify that when a user clicks on this menu item, the event is sent over to the editor pane which interprets it as a bold action and instructs the document to apply bold and finally that when the document serializes it comes out with a STRONG or B tag (depending on the user's preferences) around the text.

If you don't have a test that verifies that the message from the menu bar actually gets to the editor pane, how can you have confidence that bold works? How can you have confidence that the complex changes you're making to the document result in the right end effect when the document is serialized?

I suspect there are a couple of contributing factors to my confusion around this issue. Firstly, I work with text and frankly there is nothing less predictable and safe in software than text. It seems simple on the face of it, but there is a huge amount of complexity that goes on behind the scenes and user's absolutely demand that the editing experience is completely seamless and intuitive. There are few other environments where the number of possible program states is so incomprehensibly huge in a practical sense, where the differences really matter. On top of that, there's a ridiculous number of possible user actions that are all available at the same time and all of them interact with the program state is subtly different ways to try to best match the user's expectation.

In short, if there were any environment where you should be afraid of making changes, it's code that deals with text. That fear turns into a desire to write lots of tests and make them as close as possible to what the user is actually doing. Capturing all the subtleties of the state and embedding them in an atomic test is difficult to get precisely write – you tend to cover the most important details but miss one or two bits of state that can come back to bite you when you least expect it. Having end to end tests resolves that sense of fear, because you know that the program is operating just like when the user actually uses it – you can't have missed a detail somewhere, it's all the real deal.

The other contributing factor is that I haven't had the opportunity to work on a high quality code base that has very comprehensive atomic tests. Ephox's code base is quite old, it's mostly high quality code but it has some back alleys where ambushes lie in wait and it's well tested but mostly with integration level tests, not atomic tests. It's no surprise then that I don't have complete confidence in the atomic tests – they just don't cover enough of the application. That said, my confidence in the atomic tests is definitely growing as we add more tests and get better at knowing what to test and how to test it.

The bottom line is that now and for the foreseeable future, I'm not going to have enough confidence in the atomic tests to get rid of the slow end to end tests. However I do see it as important to improve our atomic tests and the confidence we have in them. Being able to verify that your changes haven't caused problems in 5-10 seconds by running the appropriate tests is a huge boost to productivity. Being able to have confidence that everything will work in a minute or two by running all the atomic tests is a extremely powerful too. Despite that, knowing that Bob the Builder is going to come along behind you and run a comprehensive suite of end to end tests as well is priceless.

I started off by writing an acceptance test for what I wanted and then started drilling down to what I needed, but it was leading me off into a rewrite of our list code because it was too difficult to see how to reuse the existing code for what I wanted. In the end, I decided to almost reverse refactor the existing code to extract out the logic that I needed. I say reverse refactor because instead of making the code simpler to read and understand, it was making it more complex – it really felt quite wrong to be applying the refactorings.

By the time I left this afternoon though, I'd reached the point where the logic I wanted to reuse was quite clearly separated and the design was starting to be cleaned up again. I've got some duplicate code lying around because I haven't finished cleaning up yet, but I'm quite happy with the way it's all shaping up. What has surprised me the most is how much more code I can reuse than I had expected to. Taking the refactorings one step at a time and depending on the tests to make sure I had things right has led to a much better design and a clearer path forward than I had ever thought possible.

One question that remains is what new tests I should be writing. While all the code has simply been refactored and is still covered by the old tests, the refactoring has exposed a bunch of new opportunities for atomic tests. For example, a couple of new classes have been split out – it's possible to add atomic tests for those to verify that they do the subset of the task that they claim to, no more and no less – currently we only have tests for the task as a whole. So far I don't have any fear that it's wrong so I'm not worrying about writing tests, but I'm trying to stay alert to my fear level to be sure that tests are added as soon as they provide benefit.

Firstly, if you have deployment system that needs to vary, or might in the future need to vary, based on the version of the product, then your deployment scripts have to be in source control with the product and be branched and versioned just like the actual source code. If your product just spits out a zip file that is uploaded to a web server for clients to download, you my want to separate the deployment of that zip file from the code base since it will change based on changes to the web server, not changes to the code. You should however still be able to build the zip file from scripts that are versioned with your source code.

In either case though, there's no reason that you should be deploying builds from your local machine. Having a centralized deployment machine is a good idea for many of the same reasons that having a standard integration machine – primarily to avoid the it worked on my computer issues. At Ephox, our integration machine is also our deployment machine and we've simplified deployment of a build to the web site down to a couple of clicks on a web page that the integration machine runs. Technically, we can kick off the deployment from any machine without having to get up and the actual work is handled by the deployment server.

The deployment script is home grown and very basic – it actually just runs an ant process on the server and passes through the output – but it has a couple of nice features. Firstly, it allows you to select any successful build of any of our products and deploy it all through the same interface. Secondly, once you've picked the build to deploy it shows you the list of check-in comments for the changes specific to that build. You can also get it to retrieve a subversion log of all changes between two builds.

Switching to a deployment server has had a few benefits at Ephox – firstly, it means that your local machine is free to continue work while the build uploads. Even though an upload can be done in the background, the computer is tied up doing the build and running all the tests for a while, plus the developer can't make changes to that source check out until it's done. It all becomes a hassle, so moving that over to a separate machine makes it easier to get on with the next task straight away.

More importantly though, it means that every single build that is sent out to a client is built the same way, from a completely clean check out2 and we know that every single test has been run against it and passed. We've had a lot of success with eliminating regressions in builds we send to clients and the central build server was a good way to make sure we didn't cut corners and skip the tests "just this once". These days running the tests is so in-grained in our culture we could probably lose the server and still not be tempted to cut corners.

That same deployment system can also deploy new builds to all our internal systems with a single command. Again, it's just a matter of copying files around so it doesn't need to be versioned with our source code, but it has given us a huge amount of testing and feedback because the whole company is always using the latest cutting edge build from development. It certainly keeps the engineers focussed on not breaking stuff too.

Looking back on it, for the small amount of time required to set it up and the almost non-existent maintenance required, setting up a central distribution server has been one of the biggest pay-offs we've seen for improving our development process. Every company should consider doing it.

1 – in fact, some would argue that automating deployment is part of automating the build↩

2 – we configure cruise control to completely delete the checked out source tree and check out a fresh copy to make sure it is in a pristine state. We have a faster build that runs on every commit that just does an update, but that build stops as soon as it has run all the tests, so no distribution is made.↩