The Myth of Measurable Productivity

Farnam Street Blog holds up a coding competition as evidence that:

If you want to make your computer programmers and engineers more effective give them “privacy, personal space, control over their physical environments, and freedom from interruption.”

The only trouble is, the coding competition is fatally flawed as a measure of normal developer productivity. It’s setup such that developers work on their own:

Each participant was also assigned a partner from the same company. The partners worked separately, however, without any communication, a feature of the games that turned out to be critical.

So what we actually learn is that if you’re working by yourself without interacting with anyone else, you will perform better if you don’t have anyone else around – that’s not overly surprising.

What happens in the real world however is that a team of developers work together to build software. The research is unable to shed any light on the conditions that benefit that mode of development because it measured something completely different.

And this isn’t the only example – experiments are constantly being put forward as proving or disproving things that affect productivity, but all of them are fatally flawed in one or both of two key ways:

The activity being performed in the experiment is not the same as what we’re trying to optimise in the real world.
The way they measure productivity is flawed – usually because it only measures a subset of tasks and attributes that would matter in the real world over the long term.

This shouldn’t be surprising, any repeated task is best optimised by automating it and removing humans altogether. The remaining work then is dealing with exceptions and is inherently unique (if the same exception kept happening we’d automate it). More importantly though, the idea that productivity of modern office workers can be measured is a myth. Any metrics you put around productivity are inevitably incomplete and lead to people gaming the system, working only on the measured parts of productivity and to the detriment of anything that isn’t measured. Measured productivity goes up, but those results aren’t reflected in the business’s bottom line – they’re fictional.

The only viable option is to experiment in your own workplace with your actual work and see what works for you. Use measurements as a guide but also follow some gut feel to account for the unmeasurable elements. Maybe individual offices are best for your workers, doing your work or maybe open plan is. Unless you’re in the business of completing abstract coding competitions, you probably should work that out for yourself instead of believing the research.

Continuous Integration Requires Integrating Continuously

Sarah Goff-Dupont has a post on the Atlassian blog about how Atlassian is using feature branches and still doing continuous integration:

In the not-so-recent past, continuous integration in the context of story branching was considered so impractical as to be outright incompatible. Who has time to manually configure a CI scheme for dozens of active branches that will only live a couple of days? Nobody –that’s who. And story branch-loving teams would frequently encounter *ahem* “surprises” when merging work onto the main code line –”master”, in Git-speak. But recent versions of Bamboo have essentially removed that overhead and allowed these two frenenmies to become genuinely harmonious partners in our development lifecycle.

Unfortunately, what Sarah describes is not continuous integration. At best, it’s regular integration but really it’s just automated testing. The key principal of continuous integration is that all developers are merging all changes, all the time. That way you see the messy combinatorial effects of concurrent changes and can fix them before you waste a lot of time going down incompatible paths. Anything that reduces the frequency of doing that integration is a step away from continuous integration, regardless of how often automated tests are run.

Let’s take a look at the basic process they’re using:

Next, we go into development and testing cycles on the story branch. The developer is making local commits on her workstation. [...] When she reaches a point where she wants to run tests against her work, the developer pushes those local commits up to the repository –still on the story branch, of course. Bamboo sees that there’s been a change, and starts building the corresponding Plan branch/es.

This is not continuous integration. The developer’s changes are being kept away from the main team and not integrated while they’re being developed, while they sitting just in the local git repository and while they’re in the story branch. Even though Bamboo runs tests against the code, there has been no integration done here. This is just automated testing, not continuous integration.

It seems that Atlassian have discovered this lack of integration and have developed ways of combating it:

To help find conflict before merging to master, the developer can choose to have Bamboo merge master into her story branch each time she pushes changes to the repo, then run through the CI Plan against the merged code. If the Plan passes, Bamboo will commit the merged code to her branch.

Finally we get some integration happening, but only one way. Since everyone’s off working on feature branches this is really only integrating with completed stories. All the work going on in parallel on other stories is still not integrated so we have no idea if they will work when combined.

Some teams are taking this workflow a step further and requiring developers to merge to an integration branch during development, before finally merging the completed story to master. This practice ferrets out integration issues even faster. Developers get to mash their in-progress stories together several times a day, rather than waiting until one has merged a completed story to master (which may happen only every 2-3 days) and the other has pulled code from master onto their story branch.

The key lesson here is that if we integrate more often, it’s easier to ferret out integration issues. It’s like continuous integration but with the extra fun of managing branches…

Of course, Atlassian aren’t alone in the use of feature branches – it’s become all the rage since DVCS supposedly made branching cheap and merging painless. Unfortunately while DVCS has eliminated many of the basic mechanics of merging, they haven’t done anything to reduce the risk of different developers push the codebase in incompatible directions at a structural level. The only way to avoid that is to integrate all changes as often as possible – that’s continuous integration.

The only way that feature branching and continuous integration can be at all compatible is if the build system automatically builds and tests every possible combination of branches, all the time – a massive waste of build system resources. Anything less leaves potential combinations untested until they are finally merged on to master and all the problems are discovered.

Or you know, we could just do continuous integration…

Golden Rule of Dependency Management

There’s a huge amount of complaining and problem solving for various dependency management solutions (particularly maven, but ivy and friends certainly aren’t immune). Problems range from having optional dependencies unnecessarily included to having duplicate classes or class conflicts and the solutions tend to be complex and involve a lot of trade offs. All these problems stem from breaking the golden rule of dependency management:

Own your own repository
– Sutton’s golden rule of dependency management

The vast majority, if not all, problems with dependency management comes from having incorrect, conflicting or imprecise meta-data in the repository of dependencies. Maintaining a public repository of perfectly accurate, precise and flexible dependency metadata is next to impossible – there are just too many libraries and the interrelationships are too complex. Fortunately, even extremely large companies only use a tiny subset of these libraries. With the scope reduced it’s much easier to ensure the metadata is correct and consistent.

Any time you need to introduce a new dependency, very carefully review the metadata associated it and correct any errors or inconsistencies before importing it into the repository you administer and control.

You don’t let anyone commit to your source code repository, don’t let anyone commit to your dependency repository either.

Minimise Configuration

Having configuration differences between development and production is largely unavoidable, but it’s important to keep the number of differences to a minimum to avoid unexpected bugs appearing in production that don’t occur in development mode. More than that though, it’s important to minimise configuration.

Configuration is Code

Often things are put in configuration files so that they are easy to change later, often without a full redeployment of the software, but that also implies it can be changed without going through the full testing that a release would usually be subject to. It’s rare for automated tests to cover more than one possible setting for these configuration options so changing them in production is equivalent to deploying completely untested code. What’s worse, it’s quick and easy to do with no audit trail of what happened and when.

These settings should be avoided altogether and the values hard coded in the built product and deployed along with it. That may be as simple as embedding the existing config file in the built software – the aim isn’t necessarily to mix configuration and code, only to remove the options to change the configuration outside of the normal development and deployment process.

For settings that are designed to be changed by the user as a normal part of using the software, make sure that the full range of values is tested, just like any other system input should be.

Soft Coding

The most extreme form of configuration is “soft coding” where the system becomes so configurable its essentially a poor-man’s programming language. The Daily WTF has an excellent explanation of soft coding and how even the best of intentions can lead to it.

Dealing with Production Problems

One of the biggest reason people make options configurable by administrators is so that they can quickly reconfigure things to resolve issues in production. Sadly, the worst time you could be making untested changes in production is when things are already behaving in unexpected ways. Instead, make the build, test and deployment process for your software so fast and so seamless that you can make the change in code, fully test it and get it out to production fast enough to resolve the issue. Doing so isn’t easy, but it delivers immense value beyond just resolving production issues. It allows you to more reliably and more rapidly deploy any code change into production. That means spending less time wrangling releases and more time improving the product.

Default to Development Settings

Most systems have multiple configuration profiles – one for production, one for development and often other profiles for staging, testing etc. Minimising differences between these configurations is critical but there are inevitably some things that just have to be different. This then leaves the question, what should the default settings be?

There are three main options:

Default to production values
Default to development values
Refuse to start unless an environment has been explicitly chosen

My preference is to default to development values. Why?

Development values should be “safe” in terms of any external integrations. So a developer isn’t going to accidentally start sending real buy or sell instructions to your stock broker.

There are more developers than production systems. If you default to production systems, every developer needs to remember to switch to development mode whenever they setup a new checkout. Defaulting to development mode means it just works for the most common case.

Checking authentication credentials for external systems into your source control system is generally considered a bad security practice, so the default values are unlikely to actually work in production anyway.

The down side with defaulting to development is it’s possible to accidentally deploy to production with development values causing an outage. This can be pretty easily prevented with automated deployments or using tools like RPM where files can be marked as config and thus avoid overwriting them when doing updates.

Refusing to start is the worst of all worlds – every developer has to specify a configuration mode and you still risk production outages by not specifying a mode.

Symphonious

Living in a state of accord.