I’ve been gradually tidying up the build process for UI stuff at LMAX. We had been using a mix of requires and browserify – both pretty sub-optimally configured. Obviously when you have too many ways of doing things the answer is to introduce another new way so I’ve converted everything over to webpack.

Webpack is most often used as part of a node or gulp/grunt build process but our overall build process is controlled by Buck so I’ve had to work it into that setup. I was also keen to minimise the amount of code that had to be changed to support the new build process.

Buck is a weird but very fast build tool that happens to be rather opaque about where it actually puts the things you build with it. They wind up somewhere under the buck-out folder but there’s no guarantee where and everything under there is considered buck’s private little scratch pad.

So how do you get build results out so they can be used? For things with built-in support like Java libraries you can use ‘buck publish’ to push them out to a repo but that doesn’t work for things you’ve built with a custom genrule. In those cases you could use an additional genrule build target to actually publish but it would only run when one of it’s dependencies have changed. Sometimes that’s an excellent feature but it’s not always what you want.

Symlinks have been causing me grief lately.  Our build tool, buck, loves creating symlinks but publishes corrupt cache artefacts for any build rule that includes a symlink amongst it’s output.

We also wind up calling out to npm to manage JavaScript dependencies and it has an annoying (for us) habit of resolving symlinks when processing files and then failing to find required libraries because the node_modules folder was back where the symlink was, not with the original file. Mostly this problem is caused by buck creating so many symlinks.

Probably the best thing I’ve discovered with my recent playing is Travis CI. I’ve known about it for quite some time, even played with it for simple projects but never with anything with any real complexity. Given this project uses rails which wants a database for pretty much every test and that database has to be postgres because I’m using it’s jsonb support, plus capybara and phantomjs for good measure, this certainly isn’t a simple project to test.

LMAX has invested quite a lot of time into building a suite of automated tests to verify the behaviour of our exchange. While the majority of those tests are unit or integration tests that run extremely fast, in order for us to have confidence that the whole package fits together in the right way we have a lot of end-to-end acceptance tests as well.

These tests deliver a huge amount of confidence and are thus highly valuable to us, but they come at a significant cost because end-to-end tests are relatively time consuming to run. To minimise the feedback cycle we want to run these tests in parallel as much as possible.

According to the ant documentation:

It is possible to include the same file more than once by using different prefixes, it is not possible to import the same file more than once.

Unfortunately, it turns out this isn’t _quite _true. In most cases, ant will indeed ignore a request to import a file for the second time, so:

<import>
   <file name="utils.xml"/>
   <file name="utils.xml"/>
</import>

will only import utils.xml once. This is true even if there’s a chain of files being imported (so A imports B and C, then B imports C as well, C will only be imported once).

Most people believe that ant will only ever execute a target once per execution of ant. So if we have targets A, B, C and D, where A depends on B and C and C and D both depend on D.

<target name="A" dependencies="B, C"/>
<target name="B" dependencies="D"/>
<target name="C" dependencies="D"/>
<target name="D"/>

When we run ‘ant A’, we expect each task to execute once, D first, then B  and C, ending with A. Since there is no dependency between B and C, they may execute in any order, so we might also get D, C, B, A. All pretty straight forward.

Sarah Goff-Dupont has a post on the Atlassian blog about how Atlassian is using feature branches and still doing continuous integration:

In the not-so-recent past, continuous integration in the context of story branching was considered so impractical as to be outright incompatible.  Who has time to manually configure a CI scheme for dozens of active branches that will only live a couple of days?  Nobody –that’s who.  And story branch-loving teams would frequently encounter *ahem* “surprises” when merging work onto the main code line –”master”, in Git-speak.  But recent versions of Bamboo have essentially removed that overhead and allowed these two frenenmies to become genuinely harmonious partners in our development lifecycle.

There’s a huge amount of complaining and problem solving for various dependency management solutions (particularly maven, but ivy and friends certainly aren’t immune). Problems range from having optional dependencies unnecessarily included  to having duplicate classes or class conflicts and the solutions tend to be complex and involve a lot of trade offs. All these problems stem from breaking the golden rule of dependency management:

Own your own repository

— Sutton's golden rule of dependency management

I have a bunch of JavaScript unit tests and for some of them I need to trigger a real keypress – one that activates the browser’s default functionality, not just triggers the JavaScript listeners. To do that, I have a simple signed applet that uses the AWT Robot to programmatically trigger the keypresses. Mostly, this works brilliantly.

The problem I’m having is that about 1 time in 10, the focus goes AWOL and the key events generated by the robot simply aren’t delivered anywhere. It’s generally not reliable, though I have got a setup now where I can make it happen within a few minutes of running the tests on a loop.

One of the things Ephox is contributing to TinyMCE is a build farm to run the automated tests in various browsers which winds up publishing it’s results for all to see. This has been pretty interesting to set up and there are a range of different approaches. Matt Raible posted recently about his experiences using Selenium with Sauce Labs. I had initially looked into that as well, but was worried about a few of the issues Matt hit and TinyMCE had already written a lot of tests using QUnit rather than Selenium.

Once upon a time, the state of the art for building software projects was to have a continuous integration server so your builds were repeatable, reliable and performed in a controlled environment. As a bonus they also ran a whole suite of automated tests so you knew that what came out was at least reasonable quality.

Those days have gone.

These days, it’s far more common to have a build and test farm churning away to handle the volume of tests, various different modules and projects that are all built separately and of course the requirement to run the tests on a huge variety of different platforms and environments. VMWare has been a big help in this area for quite some time, effectively letting teams build a private cloud before it was cool to call it a private cloud. Now with the growing availability of public cloud instances, it makes a lot of sense to stop buying and maintaining hardware and simply move the whole system into the public cloud.

One of the common examples of the limits of Maven and other “strict” build tools was how difficult is to concatenate CSS files together as part of the build. It’s nice to be able to split CSS files up to categorize things but the extra HTTP requests really slow down page loads, so the build should concatenate them back together. What I’ve come to realise though, is that building a custom concatenation script is just another instance where people are reinventing the wheel in build systems, and getting less power as a result.

At first glance, Ivy looks like a re-implementation of Maven’s dependency management that works nicely with ant, and to some degree it is, but it also adds some pretty significant improvements and some pretty significant complexity.

Maven Compatibility

Firstly, Ivy is compatible with Maven repositories, so if you think the way Maven manages dependencies is perfect, but don’t want to buy into the rest of Maven, Ivy provides a good answer. The configuration is a little bit different and you’ll have to learn a little bit about Ivy’s configurations which are both more powerful and more complex than Maven’s dependency “scope”, but you won’t have to go too far into them.

Here’s an important lesson for people combining ant scripts – the way basedir is calculated is very unlikely to be what you expect. In particular, if you combine the task with the task you’re probably in for a surprise.

I learnt this important life lesson when the improved build scripts I’d been working on failed on the build server even though it worked perfectly on my machine. The difference is that the build server is running cruise control and it has a wrapper ant script which checks out a fresh copy of the project then uses the task to build it. The ant task was:

Bryan comments on the three types of ant scripts:

In my experience, there are three types of Ant scripts that you encounter “in the wild”:

• Small Ant scripts, generally Java-only, which can use most of Ant’s default behaviors and are clear and simple. A lot of open source build scripts are this way.
• Serious commercial Ant scripts written before macrodef and import became available. These are generally impossible to understand and evolve, and the reality is that a small cadre of Build Wizards keep them running. Such systems often involve a substantial number of custom Ant tasks.
• Serious commercial Ant scripts written to use macrodef and import. In my experience, the need for custom Ant tasks drops way off with Ant releases post-1.6.

This really does ring true to me. Ant scripts can fairly quickly become unwieldy and difficult to work with if you aren’t using import and macrodef, but with them you can achieve so much more without the complexity getting out of hand. They won’t absolve you of the need to properly understand ant and the declarative paradigm it wants you to work with, but it’s much more approachable.

After spending some time thinking about and using different build systems, I can’t say I really like any of them all that much. I know have a reasonably complex project, using submodules that can be built with ant, buildr, maven and gradle – with varying degrees of support for the Ephox specific requirements and reporting.

Maven

Ah, the Java world’s favorite whipping boy. The more I play with Maven, the more I start to understand why it gets such a bad wrap: it’s too easy to get started and do it all wrong. Maven is not a build tool that is quick to start using – it requires planning, common conventions, a number of extras systems such as repository servers be set up and templates to be designed. However, Maven does an unfortunately good job of working if you just create a simple pom.xml and run mvn.

Having spent a bunch of time looking at various build systems and tools, one of the simplest and most effective improvements I’ve discovered is to always use the Maven project structure. It doesn’t matter if you’re not using Maven, there’s no downside to using it and every build tool that uses convention over configuration uses the Maven structure.

Previously I’ve been of the opinion that the directory structure really didn’t matter much – I went with whatever happened to be auto-generated by whatever tool I was using that day. There is a small overhead in remembering to look for the JavaSource directory instead of src or source when you switch projects but it’s incredibly minimal and not worth worrying about on its own. When it comes to build scripts, getting these basics in the same place saves a whole lot of configuration and makes everything simpler.

Recently, the subject of build tools and systems has come up again at Ephox and it appears the topic is rising up again around the internet. As part of this I’ve been reading up on and playing with a bunch of build tools to get a feel for their benefits and limitations, so it seemed worthwhile writing up what I find as I go along.

The Projects

Which build tool suits best clearly depends on the type of project you’re working with. I’m currently playing with three quite different projects: