Living in a state of accord.

Testing@LMAX – Replacements in DSL

Given our DSL makes heavy use of aliases, we often have to provide a way to include the real name or ID as part of some string. For example, an audit record for a new account might be:

Created account 127322 with username someUser123.

But in our acceptance test we’d create the user with:


someUser is just an alias, the DSL creates a unique username to use and the system assigns a unique account ID that the DSL keeps track of for us. So to write a test that the new account audit record is written correctly, we need a way to build the expected audit string.

Our initial attempt was straight forward enough:

"expression: Created account <accountId> with username <username>.",
"account: someUser",

The DSL substitutes <accountId> and <username> with the real values for us. Simple, neat, worked well for quite a while. Except that over time we kept finding more and more things that needed to be substituted and encountered situations where an audit log would reference to two different accounts or multiple instruments, leading us to have <accountId2> and <accountId3> replacements.

So a little while back some of my clever colleagues changed things around to a slightly more complex but much more flexible syntax:

"expression: Created account <accountId:someUser> with username <username:someUser>."

Essentially, the replacement now contains both the type of value to insert as well as the alias to look up. This has been a minor revolution for our DSL – it’s now much easier to handle all sorts of complex cases and it’s much clearer which alias is being used for each replacement.

The biggest win though, is that because all the required information to perform the expansions is inside the one string – not requiring any extra DSL parameters – the replacement algorithm can be easily shared across all the different places in the DSL that need to support replacements. New types of things to be replaced can easily be added in one place and are then available everyone as well.

It’s surprising how much benefit you can get from such a simple little change.

Display Port Monitor Wakes Up a Few Seconds After Being Suspending on Linux

I have a dual monitor setup with Linux where one display is connected via DisplayPort and the other via DVI, both from an Nvidia graphics card. When the screen is locked both displays go black, the DVI monitor says it’s entering power save mode and the DisplayPort monitor says “No input” at which point both monitors turn back on displaying the unlock screen.

Playing with xset dpms force standby|suspend|off gave a variety of effects, sometimes the DisplayPort stayed off but the DVI turned back on, sometimes the DisplayPort went black but didn’t turn off etc.

Ultimately though the solution was to disable the “DP Auto Switch” and “HDMI Auto Switch” setting on the DisplayPort monitor. I imagine on other monitors this is equivalent to auto detecting the input source. Now the default Linux settings do the right thing and turn both monitors off after a while, they stay off until I move the mouse and then both turn back on perfectly.

End to End Tests @ LMAX Update

A little while back I said that LMAX ran around 11,000 end to end tests in around 50 minutes. Since then we’ve deployed some new hardware to run our continuous integration on, plus continued building new stuff and are now running about 11,500 tests in under 20 minutes.

A large part of the speed boost is extra VM instances but also the increased RAM allocation available to each VM has allowed us to increase a number of limits in the system and we can now run more tests concurrently against each VM instance.

We’re currently running 61 instances of the exchange using virtual machines hosted by four Dell FX2s chassis three-quarter populated with FC630s. That gives us 480 cores and 4.5TiB RAM. That’s certainly no small investment, but we consider it excellent value for money because of the boost in productivity and confidence it gives our development team (not to mention the boost in confidence and reliability it gives our clients).

Testing@LMAX – Aliases

Even after a magnum opus on the DSL LMAX uses for acceptance tests, there’s one crucial feature that I haven’t mentioned: the use of aliases to allow tests to use simple, meaningful names while ensuring that uniqueness constraints are met.

Creating a user with our DSL looks like:


You might expect this to create a user with the username ‘user’, but then we’d get conflicts between every test that wanted to call their user ‘user’ which would prevent tests from running safely against the same deployment of the exchange.

Instead, ‘user’ is just an alias that is only meaningful while this one test is running. The DSL creates a unique username that it uses when talking to the actual system. Typically this is done by adding a postfix so the real username is still reasonably understandable e.g. user-fhoai42lfkf.

We do the same thing for instruments, venues, currencies and anything else that needs unique names.

This relatively simple trick gives us a great deal of isolation between tests that may run against the same server instance, even allowing us to run the same test multiple times without interfering with itself.