Symphonious

Living in a state of accord.

Exploring Ethereum – What’s on the Blockchain?

To understand the details of how Ethereum works I’ve been working my way through the yellow paper. And since there’s no better way to really understand things than to explain them, I thought I’d write up my own explanation for some of the things that stood out to me.

Fair warning: I’m no expert, there’s guaranteed to be sloppy usage of terminology and a high likelihood of mistakes. If you’re just looking for an easier to read version of the yellow paper, try the beige paper, it’s had some actually knowledgable people look it over. Comments and corrections are most welcome either via email or @ajsutton on twitter.

First up, what’s actually stored on the block chain? High level overviews of ethereum generally suggest that your account balance, contract code and all the data the contract stores are “on the block chain”. This is true but not in the sense that most articles would have you believe.  If you looked through the actual data for each block in the block chain you probably won’t find any of those values stored explicitly. What the block chain actually stores is the set of instructions (transactions) which when followed build up the world state that includes all those things.

At the core of an ethereum client is basically an event sourcing system which reads and applies the transactions from each block in the block chain. The world state that’s built up is then stored separately to the block chain so that a snapshot of the state is available rather than having to reprocess every transaction from scratch.  So the block chain effectively acts as the event journal for the system – it contains the instructions and all required inputs but not the actual state.

It’s important to note however, that this doesn’t mean the world state isn’t secured or reliable. Each block contains a cryptographic hash that covers the entire world state so to create a valid block you have to faithfully apply the transactions it contains because each node will verify the state hash matches.

So what is actually included in a block chain block? 

  • Block Header
    • parentHash: identifies and verifies the parent of this block
    • ommersHash: let’s skip over what ommers are for now but there’s a list of them in the body of the block and this is the hash of that list
    • beneficiary: address of the account to pay for mining this block
    • stateRoot: a hash covering the entire world state
    • transactionsRoot: a hash of the transactions listed in the block body
    • receiptsRoot: Hash of the transaction receipts generated as part of applying the transactions in this block. Note that the receipts themselves aren’t stored in the block, but this is enough to verify that every node generated the same receipts during processing.
    • logsBloom: a bloom filter of the logs generated by transactions. This allows clients to quickly establish if a particular event they’re interested in occurred in this block (with some false positives) reducing the number of blocks they’d need to get the full logs for.
    • difficulty: the difficulty level of this block (used as part of proof of work)
    • number: the number of ancestor blocks (ie: what number block is this in the chain)
    • gasLimit: Limit of gas expenditure per block
    • gasUsed: Total amount of gas used by all transactions in this block
    • timestamp: Time this block was created
    • extraData: Up to 32 bytes of extra data.
    • mixHash: part of the proof of work
    • nonce: also part of the proof of work
  • Block Body
    • List of ommer block headers: we don’t want to talk about them right now…
    • List of transactions: the actual transaction data each of which contain:
      • nonce: interesting little number taken from the senders account state. Let’s save discussing that for another day.
      • gasPrice: how much the transaction is offering to pay for gas
      • gasLimit: maximum amount of gas to spend before giving up and aborting the transaction
      • to: address of the message recipient or a 0 byte if creating a new contract
      • value: how much ether to transfer to the recipient
      • vr, and s: transaction signature data to identify and verify who sent the transaction
      • init: if creating a new contract (to is 0) this is the EVM code to execute, the result of which is stored as the new contract’s code
      • data: if not creating a new contract, this is the parameter data carried along with the message. How to interpret it is up to the contract code.

So the block body provides the journal we expected, listing each transaction (and those ommers we’re sweeping under the carpet). If we gave up decentralisation and trusted everyone, all we’d need is the transaction list and the block number to provide ordering. We’d then be left with a pretty standard event sourcing architecture. The rest, particularly all the hashes in the header, are part of reaching consensus about which chain is the right one and verifying that the transactions were applied faithfully.

One side note on contract code: since the contract code is immutable and contracts are created by transactions you might expect to find the contract code as part of the transaction data. In many cases you will, but contract creation actually happens by executing the supplied initialisation code and then storing the result that returns as the contract code. In many cases the initialisation code will effectively just return a hardcoded form of the contract code but it could do pretty much anything to generate the contract code.

Moving on from LMAX

After 7 years at LMAX, I’ll be winding up with them at the end of the month. It’s been a pretty remarkable journey and certainly not an easy decision to move on but ultimately another opportunity came up that I just couldn’t refuse.

I had intended to write up some reflections on the past 7 years at LMAX but I just can’t find the words. Suffice to say, it’s been an amazing journey full of wonderful people, big technical challenges and lots of innovative solutions. The early years in particular were just a fire hose of new knowledge to try and absorb. There have been many changes over the years, but it’s still an amazing place to work and leaving is sad.

All good things come to an end though, and I’ve got an exciting opportunity to join Consensys to do, well, pretty much anything and everything to do with Ethereum. My official title is “Blockchain Protocol Engineer” and while it’s a “distributed first” company (read I mostly still get to work from home) the team I’ll work with is mostly in Australia/NZ working on things like Ethereum clients. I’m really looking forward to being thrown back into the deep end again, delving into the Blockchain world.  I’m also looking forward to working with what seems like a team of really great people in a very rapidly growing and changing company. As an added bonus there are enough people in and around Brisbane that I’ll actually get to see my colleagues more than twice a year.

Moolah Diaries – Upgrading to Hapi 17

Hapi 17 was released a while back with quite a few nice improvement – mostly centered around using promises instead of callbacks and thus fitting in with async/await well.  However, it’s a very significant change right across it’s APIs so it took quite a while for the various plugins that Moolah uses to be updated.

Eventually though they were all ready and then the actual upgrade of Moolah code was pretty straight forward.  Moolah had been using async handlers via a plugin all along so there were minor changes to how they were specified but very few changes to the control flow they then used.

In the end it only took an evening’s work to get it all upgraded and I haven’t encountered any bugs since. That’s a pretty good vote of confidence in the effectiveness of it’s tests.

The next big upgrade will be to move the front-end over to WebPack v4. The improvements there look excellent, removing a lot of the previously required config and simplifying things but that also means it’s a pretty complex migration.

Moolah Diaries – Earmarked Funds

I dropped out of the habit of writing about Moolah but it’s been ticking along doing what I need it to for a fair while with a few additions here and there. Recently I spent a chunk of time adding the ability to ‘earmark’ money for a particular use.

My main motivation is that we’re planning a family trip to Germany at the end of the year and I want to be able to set aside money through the year as we save for it. Various things will need to be paid upfront and those expense need to come out of the earmarked funds. I could track that with a new account for the savings, but since I’m not setting up a new bank account the balances wouldn’t match up and I’d spend a lot of time transferring money in and out of the holiday account.

So I’ve introduced ‘Earmark’ as a new type of entity. In many ways they’re similar to accounts, but a single transaction can have both an account and an earmark, affecting the balance of both. So the transaction recording paying for our flights has the credit card as it’s account and the earmark set to the Germany holiday. It acts as an expense reducing the balance of both the account and the earmark.

Similarly, my wife’s income goes into our bank account and for this year I’m also earmarking it for Germany so it increases the bank balance (we did actually get the cash) and also increases the balance of the earmark.

This sounds like double counting, but the earmark balance is not included in the net worth which is still just the sum of account balances. The UI also displays “available funds” which is the sum of account balances less the sum of earmark balances.

The result of this is that an expense with an account and earmark reduces the balance of the account (we now have less money in total) but also reduces the balance of the earmark so there is less money set aside for that purpose. As a result the net worth is reduced but the available funds remains the same because the cost came out of our reserved funds.

Income with an account and an earmark increases the balance of both, so net worth goes up but available funds stays the same – those funds have been earmarked for that purpose and aren’t available for general use.

But what if we want to earmark a fixed amount per month separately to any actual income? To support that, Moolah allows creating transactions with an earmark but no account. In that case we’re recording the allocation of funds but with no actual cash movement.

The feature set is rounded out by tracking how much has been saved (total of income) and how much has been spent (total of expenses), a savings target and target date and providing some visualisations to show savings progress vs time and expense breakdowns.

The Curious Case of Vuetify Bug 2773

I ran into an issue with Vuetify recently where the 1.0 betas were constantly utilising a CPU core when run in Safari. It turned out, I wasn’t alone and it had been reported as bug 2773.

There wasn’t a lot to go on – the problem only happened in Safari and the timeline showed it was constantly invalidating and recalculating the layout, apparently without any JavaScript executing.

First step was to create a simple reproduction case, which turned out to be a template with just:

<v-app>
  <v-navigation-drawer app></v-navigation-drawer>
</v-app>

So something to do with navigation drawers and whatever effects the app attribute causes. Looking at how the generated DOM changes with and without ‘app’ led me to discover the suspicious looking CSS:

max-height: calc(100% - 0px);

Sure enough, the problem went away if that was changed to a simple max-height: 100%; but the ‘- 0px’ part isn’t as silly as it seems. If you have a toolbar at the top of the page it adjusts to allow for the toolbar height becoming ‘max-height: calc(100% – 64px)’ which also causes high-CPU. But an element with ‘max-height: calc(100% – 0px)’ by itself doesn’t trigger the bug so there must be something else contributing to the problem…

More digging ensued, including deleting large chunks of Vuetify’s css to see when the problem went away which lead to the rule:

.navigation-drawer[data-booted="true"] {
  transition: 0.2s cubic-bezier(0.4, 0, 0.2, 1);
}

That and the max-height was all the CSS required to reproduce the problem when Vuetify created the DOM, but not enough when applied to a static HTML page.

The ‘data-booted’ attribute is added dynamically by Vuetify when the component is loaded to trigger an enter animation on the element but I still couldn’t reproduce the issue without Vuetify even if I wrote JavaScript to add that attribute after a delay.

It turns out the third piece required to trigger the high CPU usage is that the element has to be created with document.createElement. If it’s included in the page HTML or even created with el.outerHTML = ‘…’ there’s no issue.  So it turns out to be a Safari/WebKit bug which can be reproduced with:

  <div id="app">
  </div>
  <script>
    const nav = document.createElement('aside');
    nav.style.maxHeight = 'calc(100% - 0px)';
    const originalEl = document.getElementById('app');
    originalEl.parentElement.replaceChild(nav, originalEl);
    requestAnimationFrame(function() {
      nav.style.transitionDuration = '1s';
    });
  </script>

You can try it out.  With Safari 11.0.2 (and possibly other versions) you can see the Timelines tab in dev tools show the page constantly recalculating layout and Activity Monitor will show the process for that tab using around 100% CPU.

The work around for Vuetify was to ensure that transition-property which was defaulting to ‘all’ (because lots of things about navigation drawers animate) doesn’t include the ‘max-height’ property or anything else that uses ‘calc’. Hence pull request 2958.

The problem appears to have already been fixed in WebKit as I can’t reproduce it in the current Safari Technology Preview build.