Living in a state of accord.

Moolah Diaries – Multi-tenant Support for Testing

Experience at LMAX has very clearly demonstrated the benefits of good test isolation, so one of the first things I added to Moolah was multi-tenant support. Each account is a completely isolated little world which is perfect for testing.

Given that I’ve completely punted on storing passwords by delegating authentication to Google (and potentially other places in the future since it’s using bell to handle third party authentication), there’s actually no user creation step at all which makes it even easier. As a result it’s not just acceptance tests that are benefiting from the isolation but also the database tests which no longer need to delete everything in the database to give themselves a clear workspace.

It’s all quite civilised really.

Moolah Diaries – Vuex for Front End State

Most of the work I’ve done on Moolah so far has been on the server side – primarily fairly boring setup work and understanding the best way to use any number of libraries that were new to me sensibly. The most interesting work however has been on the front-end. I’ve been using Vue.js and Vuetify for the UI after Vue’s success in the day job. The Moolah UI has much more data-interdependence between components than what we’ve needed at work though so I’ve introduced Vuex to manage the state in a centralised and more managed way.

I really quite like the flow – Vue components still own any state related directly to the UI like whether a dialog is shown or hidden etc but all the business model is stored in and managed by the Vuex store. The Vue components dispatch actions which perform computation, make requests to the backend or whatever is required then commit any changes to the store (via mutations). The usual Vue data-binding then kicks in to update the UI to reflect those changes.

The big advantage of this is that it naturally pulls business logic out of .vue files, preventing them getting to big. Without Vuex that basically depends on having the discipline to notice when a .vue file is doing too much and then untangling and splitting out the business logic.  Vuex provides a much clearer and more consistent way to delineate business logic from view code because you can’t modify state directly from the Vue component and it then becomes natural to split out an action.

Vuex’s module support also makes it easy to avoid your Vuex store from becoming the big ball of mud that does everything.

However, I’m still searching for a good, efficient way to calculate and update the current balance for each transaction. The actual calculation is simple enough – the balance for any transaction is the sum of the amount of every transaction before it in the account. Simplistically we could just start from the first transaction and iterate through calculating all the balances in a single O(n) pass. However, recalculating the balance for every transaction on each change is incredibly wasteful and is a big part of why the original version of Moolah takes so long to get started – it’s calculating all those balances. Each transaction balance actually only depends on two things, the transaction amount and the balance of the previous transaction. Since most new or changed transactions are at or near the very end of the transaction list, we should be able to avoid recalculating most of the balances.

I don’t think Vue/Vuex’s lazy evaluation will be able to avoid doing a lot of extra recalculation, not least of all because the only way to represent this would be a transactionsWithBalances computed view and it would output the entire list of transactions so would recalculate every balance on every change.

However, it’s reasonably straight forward to build the lazy evaluation manually, but where does that sit in the Vuex system? I’m guessing pre-calculated balance is just a property of every transaction in the state and actions take responsibility for updating any balances they might have affected.

I’m leaning towards having a dedicated ‘updateBalances’ action that can be triggered at the end of any action that changes transactions and is given the first transaction that requires it’s balance recalculated. Since every transaction after that depends on the balance of the one before they’ll also need updating.

I think that works and am now reminded about how useful it is to write a diary like this as a way to think through issues like this.

Modernising Our JavaScript – Vue.js To The Rescue

I’ve previously written about how Angular 2 didn’t work out for us, our second attempt was to use Vue.js which has much far more successful. The biggest difference with Angular is that it’s a much smaller library. It has far less functionality included in the box but a wide range of plugins to flesh out functionality as needed. That avoided the two big issues we had with Angular 2:

  • Build times were faster because there were way fewer dependencies to install, manage and bundle together. Not using Typescript may have also helped but I never had a clear indication of how much that affected build time.
  • It integrated much more easily with our existing system because it didn’t try to do everything. We just kept using the bits of our system that worked well.

We’ve been so happy with how Vue.js fits in for us, we’re now in the process of replacing the things we had built in Angular 2 with Vue.js versions.

We set out looking for a more modern UI framework primarily because we wanted the data binding functionality they provide. As expected, that’s been a very big benefit for any parts of the UI that are even slightly more than simple CRUD. We were using mustache for our templates and the extra power and flexibility of Vue’s templating has been a big advantage. There is a risk of making the templates too complex and hard to understand, but that’s mitigated by how easy it is to break out separate components that are narrowly focused.

In fact, the component model has turned out to be the single biggest advantage of Vue over jquery. We did have some patterns built up around jquery that enabled component like behaviour but they were very primitive compared to what Vue provides. We’ve got a growing library of reusable components already that all fit in nicely with the existing look and feel of the UI.

The benefit of components is so great that I’d use Vue even for very straight-forward UIs where jquery by itself could handle it simply. Vue adds almost no overhead in terms of complexity and makes the delineation of responsibilities between components very clear which leads to often unexpected re-usability. With mixins it’s also possible to reuse cross-cutting concerns easily.

All those components wind up being built in .vue files which combine HTML, JavaScript and styles for the component into one file. I was quite sceptical of this at first but Vue provides a good justification for the decision and in practice it works really well as long as you are a bit disciplined at splitting things out into separate files if they become at all complex. Typically I try to have the code in the .vue file entirely focused on managing the component state and split out the details of interacting with anything external (e.g. calling server APIs and parsing responses) into helper files.

Ultimately, it’s the component system that is really bringing us the most value which is a bit of a surprise given we had expected data-binding to be the real powerhouse. And data-binding is great, but it’s got nothing on the advantages of a clear component system that’s at just the right level of opinionated-ness for our system. We’re not only building UIs faster, but the UIs we build are better because any time we spend polishing a component applies everywhere it’s used.

I’m really struggling to think of a case where I wouldn’t use Vue now, and if I found one it would likely only be because one of the other similar options (e.g. Angular 2) was a better fit for that case.

Moolah Diaries – Principles

It’s always important to have some basic principles in mind before you start writing code for a project. That way you have something to guide your technology choices, development practices and general approach to the code. They’re not immovable – many of them won’t survive the first encounter with actual code – but identifying your principles helps to make a code base much more coherent.

Naturally for the Moolah rebuild I just jumped in and started writing code without much thought, let alone guiding principles but a few have emerged either as lessons from the first version, personal philosophy or early experiences.

Make the Server Smart

In the first version of Moolah, the server was just a dumb conduit to the database – in fact for quite some time it didn’t even parse the JSON objects, just stuffed them into the database and pulled them back out again. That missed a lot of opportunities for optimisation and keeping the client light and fast.

Ultimately this isn’t so much about client vs server, it’s more about backend vs frontend. In a browser the data storage options are limited (though much better now than 5 years ago) but the server has a proper database so can filter and work with data much faster. Sure I could re-implement fast indexing in the browser but why bother?

Provide Well-Defined, Client-Agnostic APIs

I want to be able to work on the client and server independently, potentially replacing one outright without changing the other. That way if one of my shiny tool choices doesn’t pan out well it’s a lot easier to ditch it, even if that means completely rebuilding the client or server. It also means I can play around with native applications should I want to in the future.

Deploy Early, Deploy Often and Deploy Automatically

This is basically the only sane way to work, but I’m often too lazy to do it properly with side projects. Moolah still has a few missing elements but has come far enough that I’m likely to stick with this and do it properly.

Test at Multiple Levels

Side projects that involve playing with shiny new toys often wind up lacking any form of tests and that comes back to hurt pretty quickly. I’ve made an effort to have a reasonable level of tests in place right from the start. There’s been a big cost to that as I wrestle with the various testing frameworks and understand the best ways to approach testing Moolah, but that cost will be paid back over time and is much lower than trying to wrestle with those tools and a big, test-unfriendly code base.

I’m also fairly inclined to only believe something is actually working when I have quite high level tests for it, including going right through to the real database. Unit tests are a design tool, integration tests are where you flush out most of the bugs and I’ve been seeing that even this early.

Fail Fast

If a tool, library or approach isn’t going to work I want to find out quickly and ditch it quickly. I’m trying lots of new things so have had to be pretty ruthless at ripping them back out, not just if they don’t work, but if they introduce too many gotchas or hoops to jump through.

Have Fun

There’s a degree to which I need the Moolah rebuild to reach feature parity with the old system fairly quickly so I can switch over, but this is a side project so it’s mostly about having fun. I’ve let myself get distracted with building a fancy page for before you’re logged in even though it’s way more than I’ll likely ever need.

I’m not sure this is a particularly meaningful list in general but for this particular project at this particular time they seem to be serving me well.

Moolah Diaries – Background

When I moved back to Australia about 5 years ago, I suddenly had a much more complex task of tracking our money. I was consulting, so had to set aside money to pay tax at the end of the year, we were buying our first house and having just moved countries needed to buy pretty much everything so budgeting was critical. I found a reasonable looking money tracking application and started recording all our transactions there and using it to manage our money.

That worked for a year or so but I wound up frustrated that it couldn’t produce exactly the view of data I wanted and couldn’t forecast into the future quite in the way I wanted. So I wound up writing my own because it seemed like fun and would let me play around with shiny new technologies.

I those crazy days backbone was all the rage and bootstrap was the latest must-have look so I went with those.  I also built it with the intention that it could run entirely as a standalone, offline webapp using a backbone plugin to store data in local storage. It could sync data across devices via a web server but the intention was to support going offline without any notice or loss of functionality.

I very imaginatively called my new thing “finance” because why waste time thinking of a name when there’s code to write. However, when I set up a domain to sync through StartSSL (who used to provide free, trusted SSL certs but who are no longer trusted) refused to issue a cert for a domain containing the word finance. Apparently you have to pay if you’re doing anything to do with financial transactions. I tried money and a few other variants before finally discovering that “moolah” got through the filters and so the project became known as Moolah.

As with most personal projects, it fairly quickly reached the point where it could do everything I needed, including the particular reports, graphs and views I couldn’t get from the previous software. Everything was great, and I was happy.

So I stopped changing the code.

Time moved on, no one talks about backbone anymore, bootstrap’s still doing pretty well but I never liked it much and I kept on adding all my transactions into Moolah. As the number of transactions increased, some of the underlying design decisions were challenged and all the “I’ll improve that later when I need to” things started to become needed. But it had been years since I’d worked on the code and now it was an unfamiliar, legacy code base that some imbecile had written while obviously more interested in playing with shiny new things than writing maintainable code.

Since it was designed to work offline, every transaction was sent down to the browser so it had all the information. And due to one too many annoying bugs in the backbone local storage plugin it didn’t so much sync as just download everything every time you opened the page.  The size of the data was pretty insignificant, but the number of transactions to be filtered and totalled up etc meant that eventually it just wasn’t worth trying to use on my phone, then Safari started struggling and eventually even Chrome on my very beefy dev box struggled.

I’m sure I could work through the old code, fix up a bunch of it’s poor assumptions and keep it chugging along, but this is spare-time coding and that doesn’t sound at all fun. Instead, I’m going to rebuild it from scratch using lots of shiny new technologies but without making the same mistakes as last time.

Except the one about using shiny new technologies that may or may not last. I’m totally making that mistake again because it’s just too much fun. But I will try to avoid going too crazy with it.

And since my blog is far too neglected these days, I’m going to try and write about it from time to time. We’ll see how that goes…