Living in a state of accord.

Developing First Class Software with the Developers that Google Rejects

Ron Qartel: Developing First Class Software with the Developers that Google Rejects

Focus on building a great team and a great way to develop, not on hiring individual hotshots. I would caution however that XP isn’t necessarily the “great way to develop” that will work best for your team and your circumstances, but its certainly a good starting point. Just remember that the one key Agile principal is to continuously improve the way you work. So don’t just follow “the rules”.

Continuous Integration Requires Integrating Continuously

Sarah Goff-Dupont has a post on the Atlassian blog about how Atlassian is using feature branches and still doing continuous integration:

In the not-so-recent past, continuous integration in the context of story branching was considered so impractical as to be outright incompatible.  Who has time to manually configure a CI scheme for dozens of active branches that will only live a couple of days?  Nobody –that’s who.  And story branch-loving teams would frequently encounter *ahem* “surprises” when merging work onto the main code line –”master”, in Git-speak.  But recent versions of Bamboo have essentially removed that overhead and allowed these two frenenmies to become genuinely harmonious partners in our development lifecycle.

Unfortunately, what Sarah describes is not continuous integration. At best, it’s regular integration but really it’s just automated testing. The key principal of continuous integration is that all developers are merging all changes, all the time. That way you see the messy combinatorial effects of concurrent changes and can fix them before you waste a lot of time going down incompatible paths. Anything that reduces the frequency of doing that integration is a step away from continuous integration, regardless of how often automated tests are run.

Let’s take a look at the basic process they’re using:

Next, we go into development and testing cycles on the story branch.  The developer is making local commits on her workstation.  […]  When she reaches a point where she wants to run tests against her work, the developer pushes those local commits up to the repository –still on the story branch, of course.  Bamboo sees that there’s been a change, and starts building the corresponding Plan branch/es.

This is not continuous integration. The developer’s changes are being kept away from the main team and not integrated while they’re being developed, while they sitting just in the local git repository and while they’re in the story branch. Even though Bamboo runs tests against the code, there has been no integration done here. This is just automated testing, not continuous integration.

It seems that Atlassian have discovered this lack of integration and have developed ways of combating it:

To help find conflict before merging to master, the developer can choose to have Bamboo merge master into her story branch each time she pushes changes to the repo, then run through the CI Plan against the merged code. If the Plan passes, Bamboo will commit the merged code to her branch.

Finally we get some integration happening, but only one way. Since everyone’s off working on feature branches this is really only integrating with completed stories. All the work going on in parallel on other stories is still not integrated so we have no idea if they will work when combined.

Some teams are taking this workflow a step further and requiring developers to merge to an integration branch during development, before finally merging the completed story to master. This practice ferrets out integration issues even faster. Developers get to mash their in-progress stories together several times a day, rather than waiting until one has merged a completed story to master (which may happen only every 2-3 days) and the other has pulled code from master onto their story branch.

The key lesson here is that if we integrate more often, it’s easier to ferret out integration issues. It’s like continuous integration but with the extra fun of managing branches…

Of course, Atlassian aren’t alone in the use of feature branches – it’s become all the rage since DVCS supposedly made branching cheap and merging painless. Unfortunately while DVCS has eliminated many of the basic mechanics of merging, they haven’t done anything to reduce the risk of different developers push the codebase in incompatible directions at a structural level. The only way to avoid that is to integrate all changes as often as possible – that’s continuous integration.

The only way that feature branching and continuous integration can be at all compatible is if the build system automatically builds and tests every possible combination of branches, all the time – a massive waste of build system resources. Anything less leaves potential combinations untested until they are finally merged on to master and all the problems are discovered.

Or you know, we could just do continuous integration…

Cross Pairing

This evening I stumbled across two interesting posts about alternate layouts for pairing rather than sitting side by side. Firstly Tom Dale talking about Tilde’s corner desk pairing setup (and some of the software setup they use) and that was inspired by Josh Susser’s face to face pairing setup at Pivotal Labs.

Both approaches require more floor space which makes them difficult to setup but I would expect the face to face pairing to be a heck of a lot better if done well. I’ve always preferred having separate screens in mirror configuration as well as separate keyboards and mice to allow the developers to sit a little further apart to be comfortable and to be able to look straight ahead at the screen. That said, I quite like having a second monitor for spreading out windows as we have at LMAX so it’s not clear cut which is better.

It’s also interesting to note the popularity of the flat screen iMacs as opposed to Mac Pros or laptops. The former being too expensive for extra power and extensibility that generally isn’t required and the latter often being a bit too individualised to be good as pairing machines. Plus laptops, while amazingly powerful these days still have less bang for the buck and the reduction in performance is just enough to matter for development.

Daily Deployment

One of the challenging practices in XP is daily deployment – it requires your development team to have a very low defect rate and completely automated build and deployment tools. In an off the shelf software scenario it has the additional challenge that you can't actually get your customers to upgrade every day.

At Ephox we've adhered to this practice pretty well in the past by automatically deploying builds to all our internal systems, including our corporate wiki, website CMS and even this blog. Of course, that's still just a small subset of the kind of environments that we're actually used in and it's not actually getting out to real customers. To take the next logical step, we need to make those builds available to any customers who feel like checking out what's new – maybe not every day, but between all our customers hopefully regularly enough to give us good feedback.

So that's what we've done with the EditLive! Early Access program on LiveWorks! If everything is going well and the engineers are actually working on EditLive! (they do like to go home on weekends) a new build should be automatically deployed to LiveWorks! twice a day ready for people to try out. We also include the current change log on the page (again, automatically updated) but at this stage we're only listing changes from the last released version rather than every little change we make in the change log so it doesn't necessarily show changes in every build. We might need to look into publishing commit logs as well in the future to work around that.

Personally, I think this is pretty exciting – there's a lot of cool stuff that gets developed early in the release process that previously hasn't seen the light of day until months later. Now it's up on LiveWorks! within 12 hours.

As always, if you have any problems, suggestions random doodlings on the matter, comments are below, my contact details are just over there on the right and there's also the LiveWorks! mailing list.

Doug’s Excited…

So Doug is excited about how we took our first steps in a new product and how well it went. Personally, I'm impressed with the way that we presented all the usual engineering setup tasks to the client in a client focussed manner. We could have done it better by not discussing up front all the engineering tasks we were hiding behind the suggested first story, but that's okay. The first story was that we wanted to ship a distribution of the new product. It's really quite backwards to think of things that way – normally you build the product then work out how to package and distribute it, but it's impossible to ship anytime if you don't build the distribution mechanism at the start.

So hidden in that story was the following engineering tasks:

  • Create a subversion repository.
  • Create a build process (including managing build numbers, version numbers etc).
  • Create a change log.
  • Create zip file for distribution.
  • Add the project to cruise control so it is automatically built.
  • Adjust our "AutoDeployer" to be able to deploy the built zip file to the web site.

We also found time to pay back a bunch of technical debt left around our systems:

  • Cleaned up some cruft in our Subversion repository that we don't use anymore.
  • Fixed the build process for our IWWCM integration so it runs about 10 times faster and builds only trigger when appropriate – they were triggering on unrelated changes in subversion.
  • Cleaned up and improved the AutoDeployer.

Not to mention, we've learnt a lot about starting new products from scratch and gotten that process up and running quite well. I guess you could say I'm excited too.