Ron Qartel: Developing First Class Software with the Developers that Google Rejects

Focus on building a great team and a great way to develop, not on hiring individual hotshots. I would caution however that XP isn’t necessarily the “great way to develop” that will work best for your team and your circumstances, but its certainly a good starting point. Just remember that the one key Agile principal is to continuously improve the way you work. So don’t just follow “the rules”.

Sarah Goff-Dupont has a post on the Atlassian blog about how Atlassian is using feature branches and still doing continuous integration:

In the not-so-recent past, continuous integration in the context of story branching was considered so impractical as to be outright incompatible.  Who has time to manually configure a CI scheme for dozens of active branches that will only live a couple of days?  Nobody –that’s who.  And story branch-loving teams would frequently encounter *ahem* “surprises” when merging work onto the main code line –”master”, in Git-speak.  But recent versions of Bamboo have essentially removed that overhead and allowed these two frenenmies to become genuinely harmonious partners in our development lifecycle.

This evening I stumbled across two interesting posts about alternate layouts for pairing rather than sitting side by side. Firstly Tom Dale talking about Tilde’s corner desk pairing setup (and some of the software setup they use) and that was inspired by Josh Susser’s face to face pairing setup at Pivotal Labs.

Both approaches require more floor space which makes them difficult to setup but I would expect the face to face pairing to be a heck of a lot better if done well. I’ve always preferred having separate screens in mirror configuration as well as separate keyboards and mice to allow the developers to sit a little further apart to be comfortable and to be able to look straight ahead at the screen. That said, I quite like having a second monitor for spreading out windows as we have at LMAX so it’s not clear cut which is better.

One of the challenging practices in XP is daily deployment – it requires your development team to have a very low defect rate and completely automated build and deployment tools. In an off the shelf software scenario it has the additional challenge that you can’t actually get your customers to upgrade every day.

At Ephox we’ve adhered to this practice pretty well in the past by automatically deploying builds to all our internal systems, including our corporate wiki, website CMS and even this blog. Of course, that’s still just a small subset of the kind of environments that we’re actually used in and it’s not actually getting out to real customers. To take the next logical step, we need to make those builds available to any customers who feel like checking out what’s new – maybe not every day, but between all our customers hopefully regularly enough to give us good feedback.

So Doug is excited about how we took our first steps in a new product and how well it went. Personally, I’m impressed with the way that we presented all the usual engineering setup tasks to the client in a client focussed manner. We could have done it better by not discussing up front all the engineering tasks we were hiding behind the suggested first story, but that’s okay. The first story was that we wanted to ship a distribution of the new product. It’s really quite backwards to think of things that way – normally you build the product then work out how to package and distribute it, but it’s impossible to ship anytime if you don’t build the distribution mechanism at the start.

I’ve seen a misconception a number of times where people believe that the regular release cycles and TDD practices in XP mean that you don’t need a standard QA process – that the developers are responsible for writing perfect code, first time every time. It’s true that XP practices can significantly improve quality, both of the code and the final product, but that doesn’t excuse the team from properly testing their work.

One of the really core principles of XP is the idea of small, frequent releases. The most obvious benefit of this is that it allows users to see the product and provide feedback about what could be done to improve it. What’s not so obvious, but is part of the reasoning behind a number of XP principles, is that the developers benefit from this just as much, if not more than the users. Being couped up in an office somewhere slaving away creating software that no one’s using just isn’t fun. Getting your software out to users and having the feeling that you’ve achieved something is a big feel-good moment and it energizes the development team, allowing them to keep going full pace instead of getting tired and stressed.

Slashdot linked off to an old interview with Kent Beck and Cynthia Andres, but it was interesting to see the number of people complaining about the pair programming aspect of XP. Comments like:

Programmers are solo beasts – putting two of these dragons behind one keyboard is asking for trouble.

and:

In the OO Programming course I had this year, they encouraged us to practice extreme programming. Well, it sucked… Everytime my partner and I encountered a little problem and one of us had an idea, he had to explain it to the other, which took at least 3 times as long as just typing it out, which in almost every case made the idea perfectly clear. Suffice to say, after about 4 hours we gave up and just sent each other any files we had edited (of course making sure we didn’t work on the same file at the same time).

There’s an interesting pattern when you start doing TDD and trying to make your tests as atomic as possible1. First of all you wonder how anyone could ever get far with completely standalone classes that don’t interact with anything – obviously a program needs some level of communication between classes. Then you discover Mock objects. These wonderful little gems allow you to have communication between classes but still test them independently. Pretty soon you’re going on a cookie monster style binge session with mocks. Everything can should and will be mocked out and there’s no longer any need to worry about making your classes keep to themselves, all those external dependencies can just be mocked out.

At least in my mind, there seems to be a clash of aims in XP. You want to make sure that you have complete confidence in your tests so that you can go faster and reduce the cost of change. To achieve this you write lots and lots of tests – until your fear of something breaking turns to boredom from writing tests you know will pass. Most of those tests are atomic and test a particular component, but fear lies in the gaps between components too so you regularly get recommendations like Ola Ellnestam’s on my previous post, Testing Your Setup Code:

I’ve had an interesting experience the last couple of days – I’ve been trying to add some major new functionality into our list code. The code is exceptionally well tested and fairly easy to understand but it wasn’t clear how to write a test that described the functionality I wanted to add.

I started off by writing an acceptance test for what I wanted and then started drilling down to what I needed, but it was leading me off into a rewrite of our list code because it was too difficult to see how to reuse the existing code for what I wanted. In the end, I decided to almost reverse refactor the existing code to extract out the logic that I needed. I say reverse refactor because instead of making the code simpler to read and understand, it was making it more complex – it really felt quite wrong to be applying the refactorings.

Once you have an automated build, the next step is to automate deployment1. A lot of people take this to mean that you should be able to check out the code, compile it and deploy it all from your local work station. I think this is largely a really bad idea.

Firstly, if you have deployment system that needs to vary, or might in the future need to vary, based on the version of the product, then your deployment scripts have to be in source control with the product and be branched and versioned just like the actual source code. If your product just spits out a zip file that is uploaded to a web server for clients to download, you my want to separate the deployment of that zip file from the code base since it will change based on changes to the web server, not changes to the code. You should however still be able to build the zip file from scripts that are versioned with your source code.

Having a fast build is an important part of XP so that the continuous integration cycle doesn’t take too long. Do people usually see this as including all the acceptance tests as well or does this just include the developer’s tests?

How do people handle acceptance tests failing? Does it cause the build to fail or not? What about acceptance tests that have been added for the current iteration but haven’t been implemented yet?

There’s a lot of talk about how you convert a legacy project over to TDD, and there’s a lot of talk about how you TDD on little code examples, but how do you get started on a real project? For a swing app, how do you TDD public static void main(String[] args)?

Atomic testing wants you to be able to mock out your dependencies, which leads to things like dependency injection. Somewhere along the line though, someone has to create those objects and pass them in. How do you atomically test that?

In James Shore’s chapter on Continuous Integration for his upcoming book, he strongly recommends using synchronous integration, where each time you check in you stop and wait for two builds to complete before moving on, in favor of asynchronous integration, where you check in and move on to your next task leaving an automated build server to check everything’s still okay. The whole point of continuous integration is that your team integrates regularly, so that you avoid conflicts. I can’t see how this is encouraged by adding up to twenty minutes of work1 every time you want to check in.

I just stumbled across something interesting that I probably should have realized before. With TDD, the first thing you should do when you get assigned a bug to fix is write a test that reproduces it. This morning I was good and did just that, but surprisingly the test passed.

Now normally when your test passes straight off, it means you didn’t write it correctly and you need to fix the test. In this case though I could confirm manually that the bug simply didn’t happen – a classic works for me. Drop back into bugzilla and mark the bug WFM and I suddenly realized I could prove that it works correctly by pointing to the test.

One of the most common things touted about XP is that it allows for rapid change. The client can change the requirements on the fly for a very low cost – they can add features, changes features and drop features regularly as the product is developed without incurring a massive cost of change. When you work on in-house applications or custom built software, that provides a really big benefit, but when you develop off the shelf software that means that the final release may contain a mish-mash of new features and be generally unmarketable.

In my previous post on tracking estimate accuracy, Stephen Thorne commented:

I thought the entire idea about estimates is that they should be self correcting? Estimates made in some arbitary numbering system, you sum up the estimates, divide by the time actually taken, then you have an accurate picture of how one developers estimates map to reality…

The ability to map arbitrary estimates to actual time that Stephen refers to is velocity. Essentially, how many points per time period can the team complete? This is a really effective technique if your estimates are consistent. That is, you may find that 3 points takes on average, a day to complete. So your velocity is 3 points per day1. However, if your estimates aren’t accurate, you may have some 3 point stories take 15 minutes, and some take a week.

Over the past few weeks we’ve been working on tracking our velocity and trying to get a grip on how many points a week we can do. We haven’t really rolled out proper iterations yet, but we pick a set of stories to work on for the week and then set about doing them. At the end of the week we add up how many points we completed. This has been working quite well except that we are regularly doing more points than we expected but not nessecarily completing any extra stories.

Feedback is a key concept in XP and it’s common for engineers to focus on feedback from the business in terms of what features are required, how to make things better etc. It is important however not to forget that feedback to the business is just as important

The business needs to know how things are going – to see that progress is being made and what the team is working on. To that end, we’ve started providing a whole series of pretty graphs showing our velocity on each project, the number of open bugs and a few graphs showing basic code quality metrics. It remains to be see how successful this will be but hopefully it will help provide some vital communication from engineering back to the rest of the business without needing an all-hands meeting every week.

XP is a process that requires constant communication and a strong commitment to making XP work. While you can do XP subversively without getting the rest of the business to buy into the process, the benefits you get will be significantly diminished. For a software development company, adopting XP affects every single part of the business and it’s important that everyone understands their role in the process and wants to help make it succeed.

All of the writings on XP that I’ve seen seem to assume that a development team only ever works on one project at a time – there’s one product to develop and deploy, one client, one timeline etc. Oh what a nice world that must be to live in.

At Ephox we work on a large number of products - all centered around the same core editor – and each product has it’s own requirements, clients and timelines that we have to satisfy. So how do you juggle these different projects so that you make optimal use of your time and keep all the products moving forward?

Greg picks up on my previous post about XP principles and how it helps avoid rewrites. I thought I should explain in more detail why rewrites are a bad thing and my thoughts about when and how you should do them anyway.

Programmers for some reason seem to think any code they didn’t write, and often any code they wrote some time ago, is poor quality, misguided and generally crap. Often this is quite true, but the degree to which the code is bad is usually significantly less than the initial impression it gives. That is, when you first look at a piece of code and start working your way through it, you feel as if the programmer was completely brain-dead and it’s amazing the software worked at all.  Except in very rare occasions the code does work with just minor bugs or even no bugs (at least that have been discovered) so the inclination to think that it can’t possibly work is just a form of panic reaction from your brain while it struggles to comprehend the new code. It’s very easy to condemn a piece of code in those first few moments when you don’t actually understand it and are just seeing a mess of symbols and a bunch of bad coding practices, but doing so condemns your rewrite.

A while back Ben Hyde wrote his thoughts on the key XP principles in What every you doing, it’s wrong! I’m not sure I fully comprehend exactly what Ben is trying to say but a lot of it seems opposed to the way I see XP in theory and to the experience I’ve had in implementing XP.

I’ll start with Ben’s rewording of the summary of Extreme Programming (see also James Mead’s original):

I realized today that I hadn’t made explicit the difference in my mind between features and stories and it’s an important difference. Essentially, a feature is a group of stories that are related and deliver a package of functionality that end users would generally expect to get all at once. For instance, inline table resizing is a feature (note: this is the ability to drag to resize tables, rows and columns – try it in Word). In the first pass, you’d probably have a single story for inline resizing of tables, but it would be too big to estimate. So you break it down into three stories, resize columns, resize rows and resize the table itself.

XP has the concept of keeping the code base in a state that you can ship at any time. That seems like a good idea as it allows you to quickly ship a new version to preempt a competitor’s release or when marketing suddenly wants to attend a trade show and have something new to talk about. In other words, allow the business requirements to drive the release schedule instead of it being purely driven by the engineering team’s schedule.

We’ve adopted a point based estimating system as part of rolling out XP, with 1 point being simple, 2 moderate, 3 difficult and 5 meaning too hard to estimate (either too big or too technically difficult). One large feature for the next release has been plagued with 5 ratings, both because the feature is big and hard to break down and because it’s technically very difficult so we’re not always sure parts are possible, let alone how long they’ll take.

We’ve been switching pairs a lot in an effort to try to share knowledge between team members before and it’s been working really well. The downside to it though is that since the team owns the tasks and people swap between tasks so much (which is what gives the shared knowledge), there’s always the danger that you will get off track and start wasting time writing code that isn’t actually needed for the user story.

In order to write atomic tests for classes I’ve gotten into the habit of injecting dependencies instead of creating them inside the class. This makes it simple to mock the dependencies but means that those dependencies are now externally visible instead of being neatly encapsulated and it means that somewhere in your code, there has to be a place that actually creates all these instances and passes them in.

I have no idea how to test that bit of code.

If you want to reap the benefits of pairing in terms of sharing knowledge around your team, you need to make sure you change pair regularly. We’ve been working on the basis that one pair works on a story until it’s done – mostly because we only have a small number of engineers and a large number of projects that we need to work on.

The problem is, we’re not sharing knowledge enough. Instead of one person knowing about that new section of code, now two do, but the rest of the team still doesn’t so we tend to want to get one or both of those two to fix any issues that come up or add any improvements. It’s not a team ownership thing anymore, it’s a pair ownership.

Setting the system look and feel is a common task in applications and if you’re shooting for 100% test coverage, it’s problematic because it uses two static methods and has checked exceptions. Typically you have something like:

void setNativeLookAndFeel() {

    try {

        UIManager.setLookAndFeel(

        UIManager.getSystemLookAndFeelClassName());

    } catch (Exception e) {

        // Handle exception somehow.

    }

}

As far as I know, there’s no way to write a test that doesn’t also exercise UIManager, so we’ll have to live with that, but we can test that the UI is set and the exception handled.

The basic principle of TDD is that you write the test first and then write the code required to make that test pass. There’s more to it though if you expect to see all the benefits. If you drive your development from integration tests instead of atomic tests, you’ll wind up with highly coupled code again and because it grows gradually without up-front planning, it’s often harder to understand.

If however, you drive your development by atomic tests, you are very strongly encouraged to decouple your code whenever possible because mocking out dependencies is so annoying and time consuming (plus the more you have to mock, the more brittle your tests tend to be).

There is one major problem with test driven development – it makes it easy to get lost. When you don’t use TDD, you tend to run your application a whole lot to see what effect your latest code has made. With test driven development, you run your atomic tests a whole lot instead. The trouble is, if your unit tests are taking you in the wrong direction, it can be a long time before you find out.

I’ve been developing some custom views (javax.swing.text.View subclasses) in the last few days. These are right up the alley of GUI code that people tend to find difficult to test but I’ve been pushing forward with test driven development with a fair bit of success. The main approach I’ve been using is to mock out pretty much everything the view interacts with and test it’s logic more than it’s actual rendering. In fact, this particular view doesn’t do any rendering of it’s own, but it does do a lot of layout of child views etc.

Most code bases have some kind of code debt associated with them, legacy code bases tend to have lots of code debt. The good news is, paying back that code debt isn’t a complete time sink – it has definite advantages too. Obviously, reducing code debt tends to make the team go faster, but it also tends to fix bugs.

Quite often where there is debt, there is not just a lack of maintainability but actual bugs that are frustrating your users. By cleaning up the code, you will often fix those bugs without any special effort to do so – it just happens because of the simplified design.

Often when you define an interface there is a series of generic tests which can be applied to every implementation of that interface. For instance, the java.util.List interface carefully defines the behavior that all implementations must have. A simple way of testing these behaviors are present is to use JUnit’s ability to inherit test methods.

Define a base class with a test method for what you want and an abstract method to create the actual instance:

Refactoring without tests is hard and really, really scary but sometimes you’ve inherited (or perhaps written yourself) a big ball of untestable mud and the only way out is to refactor and make it testable. One of our engineers found himself in that situation today so we sat down together to try and make sense of it.

I tried to follow a few simple rules:

1. Go slow. No big sweeping changes. Keep the way it does things the same and just break the code down to make it more readable and create multiple small balls of mud instead of one big one.
2. Use the automated refactorings in the IDE unless you absolutely have to make manual changes. We discovered a number of bugs in Eclipse this way where it didn’t detect identical code – who knows why not. In once case I managed to convince it the code was identical by repeatedly trying the extract method until it picked up the duplicate block but in a second case I had to revert to manually replacing the duplicate code.
3. Use the simplest refactoring that could possibly make it testable. Mostly this meant applying extract method with a smattering of extract local variable to pull out the differing pieces so duplicate code could be merged. In hind-sight I may have been better off leaving the duplicate code in and writing tests for it separately, it would then have been easier and safer to remove the duplication.

All in all I think it went really well, slow going but effective and we now have half as much code and it’s much easier to test. There’s still a lot more work needed to stop that code from smelling but it’s a start and that’s enough for one day. You can’t spend all your time refactoring legacy code, new features have to be added – just make sure that you’re constantly improving and eventually you’ll get there.

Doug suggests that XP isn’t a solution to your problems. Mostly I agree, however XP does make you feel bad – and in this case, feeling bad is good. As Doug points out, XP brings things out into the open which makes people feel bad about them. The lack of tests for your product – oh, that’s not good. Not doing continuous integration – tut tut tut. Your build is regularly broken? Shame on you! Feeling bad about these things is good and will encourage you to fix them.

There’s a big difference between code you write when you’re fresh and code you write when you’ve just had enough. I obviously had a relaxing weekend, because today my approach to coding was surprisingly well behaved – even when I had to modify legacy code that had no test coverage I made sure I wrote new tests before I touched it. Getting into TDD for new code is one thing, building up the discipline to add tests to existing code before making even a simple change is really tough.

As you spend more and more time developing good unit tests as part of TDD, you regularly find yourself writing the same thing time and time again. Fortunately, TDD also includes a refactor step to remove this duplication again. I’ve found it quite interesting that over a few weeks of this, we’ve actually developed a complete language for asserting the correct output. For example:

assertStartTagSpec(HTML.Tag.HTML);
assertStartTagSpec(HTML.Tag.BODY);
assertStartTagSpec(HTML.Tag.P);
assertContent("Hello World!");
assertLineEnding();
assertEndTagSpec();
assertEndTagSpec();
assertEndTagSpec();
assertEndOfSpecs();

Even without the domain knowledge, it should be fairly obvious what is expected. It’s definitely a lot clearer than:

It seems I finally pestered Doug into blogging again, so for his side of the story check out Experiments in Software Design.

Bob is our build server – just like the children’s TV character, he spends all day building stuff. Bob has to be one of the most beneficial improvement projects we’ve taken on at Ephox in quite some time. Every build of our key products that we ship both as full releases and hot fix patches are built by Bob. He even uploads them to the web server when told to make them available to clients.

One of the tenets of XP is that you work at a sustainable pace. Mostly that means that you don’t work such long hours that your brain turns to mush and you start churning out bad code, however there is more too it.

Working at a sustainable pace also means knowing when to take a break from a task. After weeks of slogging away at replacing this 3rd party library – constantly making progress and getting small wins from each acceptance test we got to pass – this afternoon both Doug and I seemed to have stopped caring as much as we should, so we took the afternoon off of that project and went and caught up on the other things we needed to do. When we return to working on it we should be refreshed and able to approach the problems with a more open, creative mind and thus write better code.

Doug and I are currently busy replacing a third party library which we’ve used in our product for quite some time. Thus, the first thing we need to do is create a new component which behaves in exactly the same manner as the original component. Once we have that we can start adjusting the code to add features and work in the way we really want.

To ensure we get an exactly compatible replacement, we started out by creating a suite of automated acceptance tests against the original implementation and they all passed.  Then we removed the third party library and they all failed. From there we could gradually begin building our new component by attempting to make each acceptance test pass – one at a time.

Claim: TDD Generates High Quality, Maintainable Code

Proponents of TDD argue that by writing the tests first, you are forced to focus on how the component will be used and thus will come up with a better design. My experience has actually been the opposite; TDD encourages you to just do whatever needs to be done to make the test pass and not think about design at all. I’ve never copied and pasted code so much in my life and the number of times I’ve written code that’s so messy I feel like I need a shower is getting scary. That’s okay though, it’s meant to be that way. TDD encourages short-sightedness, short-cuts and general sloppiness and that’s a good thing.

So we’ve got a huge chunk of legacy code (making up 3 main products and a lot of integration products around them), automated builds after every check-in and a bunch of tests that are somewhere between unit tests and integration tests and a team of developers who haven’t had any significant experience with XP (excepting our new hire, Doug South, who’s to blame for the XP idea in the first place). Did I mention tight deadlines and ferocious competition? We don’t have time for down-time, we need to get this process up and running and quickly and smoothly as possible.

The Ephox engineering team has decided to start adopting XP, mostly to help improve quality but also to help respond more rapidly with the changing priorities and requirements that commercial software tends to live with. I’m hoping that I’ll be able to keep a fairly good record of the process here so that over time I can look back at the challenges we’ve faced and see how we’re doing. I’m also hoping that by detailing my thoughts and experiences here that the lazy-web will chime in with useful comments, corrections and ideas.