Living in a state of accord.

Exploring Ethereum – Account and Transaction Nonce

This is the second article on things I found particularly interesting in the Ethereum yellow paper.  The first is “What’s on the Blockchain?” and the same disclaimers apply: I’m no expert and you should go verify any claims I’m making before depending on them. Comments and corrections are most welcome either via email or @ajsutton on twitter.

One of the little details in the way Ethereum works is the idea of a “nonce” attached to each account and transaction. It’s a small but important detail.

For a “normal” account (ie has no code attached), the nonce is equal to the number of transaction sent from it. In the case of contracts (accounts with code) the nonce is the number of contract-creations made by the account.

When a transaction is created, the current nonce value from the account is assigned as the transaction nonce. Part of the initial tests for intrinsic transaction validity is that the transaction nonce is equivalent to the sender account’s current nonce.

The nonce is primarily included in transactions to prevent same-chain replay attacks on transactions. The transaction sender is identified by the signature they add to the transaction (those v, r and s items from each transaction we skipped over last time). To generate those you need the account’s private key so only the account owner can create a validly signed new transaction.

However, if the transaction data and the sender are the same, the signature will also be the same. So absent the nonce, an adversary could take any existing transaction and resend it to a node with a valid signature and have it processed a second time. For example, if Alice signed a transaction to send 10ETH to Bob, Bob could take that transaction signed by Alice and repeatedly submit it for processing until all of Alice’s funds had been transferred to Bob. Bob couldn’t change anything about the transaction but that doesn’t make Alice feel any better about losing all her ETH when she only approved a single transfer.

With the account nonce however, when the transaction is first processed, Alice’s account’s nonce is incremented and then when Bob resubmits the transaction, it is rejected because the nonce doesn’t match. Bob is unable to change the nonce on the transaction without invalidating Alice’s signature so the transaction can only be applied once, exactly as Alice intended.

BUT! This doesn’t entirely eliminate replay attacks.  The transaction could still be replayed on a different chain (though it may require replaying a number of transactions so the account nonce “catches up”).  The Ethereum / Ethereum Classic split caused quite a few headaches in this regard, until EIP-155 was implemented to include an ID for the chain in the data to sign, thus making the two different chains incompatible. The same problem can also occur between test chains and MainNet, though hopefully you aren’t sharing a single private key between them.

Interestingly, most explanations for the importance of the nonce suggest it’s there to prevent double spending which is not the case. The theory goes that Alice sends transaction t1 to pay Bob for some goods but then very quickly submits another transaction t2 with a higher gas price which is then prioritised higher and mined first allowing her to spend funds twice. Even if t2 was processed before t1, it would result in Alice’s balance being reduced before t1 was applied. If there were then insufficient funds t1 would be rejected. If you had already released the goods t1 was intended to pay for that might be bad, which is why typically people wait for the transaction to be in a block at a certain depth before considering it finalised. The nonce doesn’t help prevent this double-spend issue at all – Alice could deliberately setup the same race by giving both t1 and t2 the same nonce.

Finally, the account nonce is also used as part of creating the address for a new account/contract. The address of the new account is “the rightmost 160 bits of the Keccak hash of the RLP encoding of the structure containing only the sender and the account nonce”. Which is to say, the new address is a particular way of hashing the combination of the sender’s account hash and nonce. Since the sender account nonce is incremented when sending a new transaction this is guaranteed to generate a unique address.

Exploring Ethereum – What’s on the Blockchain?

To understand the details of how Ethereum works I’ve been working my way through the yellow paper. And since there’s no better way to really understand things than to explain them, I thought I’d write up my own explanation for some of the things that stood out to me.

Fair warning: I’m no expert, there’s guaranteed to be sloppy usage of terminology and a high likelihood of mistakes. If you’re just looking for an easier to read version of the yellow paper, try the beige paper, it’s had some actually knowledgable people look it over. Comments and corrections are most welcome either via email or @ajsutton on twitter.

First up, what’s actually stored on the block chain? High level overviews of ethereum generally suggest that your account balance, contract code and all the data the contract stores are “on the block chain”. This is true but not in the sense that most articles would have you believe.  If you looked through the actual data for each block in the block chain you probably won’t find any of those values stored explicitly. What the block chain actually stores is the set of instructions (transactions) which when followed build up the world state that includes all those things.

At the core of an ethereum client is basically an event sourcing system which reads and applies the transactions from each block in the block chain. The world state that’s built up is then stored separately to the block chain so that a snapshot of the state is available rather than having to reprocess every transaction from scratch.  So the block chain effectively acts as the event journal for the system – it contains the instructions and all required inputs but not the actual state.

It’s important to note however, that this doesn’t mean the world state isn’t secured or reliable. Each block contains a cryptographic hash that covers the entire world state so to create a valid block you have to faithfully apply the transactions it contains because each node will verify the state hash matches.

So what is actually included in a block chain block? 

  • Block Header
    • parentHash: identifies and verifies the parent of this block
    • ommersHash: let’s skip over what ommers are for now but there’s a list of them in the body of the block and this is the hash of that list
    • beneficiary: address of the account to pay for mining this block
    • stateRoot: a hash covering the entire world state
    • transactionsRoot: a hash of the transactions listed in the block body
    • receiptsRoot: Hash of the transaction receipts generated as part of applying the transactions in this block. Note that the receipts themselves aren’t stored in the block, but this is enough to verify that every node generated the same receipts during processing.
    • logsBloom: a bloom filter of the logs generated by transactions. This allows clients to quickly establish if a particular event they’re interested in occurred in this block (with some false positives) reducing the number of blocks they’d need to get the full logs for.
    • difficulty: the difficulty level of this block (used as part of proof of work)
    • number: the number of ancestor blocks (ie: what number block is this in the chain)
    • gasLimit: Limit of gas expenditure per block
    • gasUsed: Total amount of gas used by all transactions in this block
    • timestamp: Time this block was created
    • extraData: Up to 32 bytes of extra data.
    • mixHash: part of the proof of work
    • nonce: also part of the proof of work
  • Block Body
    • List of ommer block headers: we don’t want to talk about them right now…
    • List of transactions: the actual transaction data each of which contain:
      • nonce: interesting little number taken from the senders account state. Let’s save discussing that for another day.
      • gasPrice: how much the transaction is offering to pay for gas
      • gasLimit: maximum amount of gas to spend before giving up and aborting the transaction
      • to: address of the message recipient or a 0 byte if creating a new contract
      • value: how much ether to transfer to the recipient
      • vr, and s: transaction signature data to identify and verify who sent the transaction
      • init: if creating a new contract (to is 0) this is the EVM code to execute, the result of which is stored as the new contract’s code
      • data: if not creating a new contract, this is the parameter data carried along with the message. How to interpret it is up to the contract code.

So the block body provides the journal we expected, listing each transaction (and those ommers we’re sweeping under the carpet). If we gave up decentralisation and trusted everyone, all we’d need is the transaction list and the block number to provide ordering. We’d then be left with a pretty standard event sourcing architecture. The rest, particularly all the hashes in the header, are part of reaching consensus about which chain is the right one and verifying that the transactions were applied faithfully.

One side note on contract code: since the contract code is immutable and contracts are created by transactions you might expect to find the contract code as part of the transaction data. In many cases you will, but contract creation actually happens by executing the supplied initialisation code and then storing the result that returns as the contract code. In many cases the initialisation code will effectively just return a hardcoded form of the contract code but it could do pretty much anything to generate the contract code.

Moving on from LMAX

After 7 years at LMAX, I’ll be winding up with them at the end of the month. It’s been a pretty remarkable journey and certainly not an easy decision to move on but ultimately another opportunity came up that I just couldn’t refuse.

I had intended to write up some reflections on the past 7 years at LMAX but I just can’t find the words. Suffice to say, it’s been an amazing journey full of wonderful people, big technical challenges and lots of innovative solutions. The early years in particular were just a fire hose of new knowledge to try and absorb. There have been many changes over the years, but it’s still an amazing place to work and leaving is sad.

All good things come to an end though, and I’ve got an exciting opportunity to join Consensys to do, well, pretty much anything and everything to do with Ethereum. My official title is “Blockchain Protocol Engineer” and while it’s a “distributed first” company (read I mostly still get to work from home) the team I’ll work with is mostly in Australia/NZ working on things like Ethereum clients. I’m really looking forward to being thrown back into the deep end again, delving into the Blockchain world.  I’m also looking forward to working with what seems like a team of really great people in a very rapidly growing and changing company. As an added bonus there are enough people in and around Brisbane that I’ll actually get to see my colleagues more than twice a year.

Moolah Diaries – Upgrading to Hapi 17

Hapi 17 was released a while back with quite a few nice improvement – mostly centered around using promises instead of callbacks and thus fitting in with async/await well.  However, it’s a very significant change right across it’s APIs so it took quite a while for the various plugins that Moolah uses to be updated.

Eventually though they were all ready and then the actual upgrade of Moolah code was pretty straight forward.  Moolah had been using async handlers via a plugin all along so there were minor changes to how they were specified but very few changes to the control flow they then used.

In the end it only took an evening’s work to get it all upgraded and I haven’t encountered any bugs since. That’s a pretty good vote of confidence in the effectiveness of it’s tests.

The next big upgrade will be to move the front-end over to WebPack v4. The improvements there look excellent, removing a lot of the previously required config and simplifying things but that also means it’s a pretty complex migration.

Moolah Diaries – Earmarked Funds

I dropped out of the habit of writing about Moolah but it’s been ticking along doing what I need it to for a fair while with a few additions here and there. Recently I spent a chunk of time adding the ability to ‘earmark’ money for a particular use.

My main motivation is that we’re planning a family trip to Germany at the end of the year and I want to be able to set aside money through the year as we save for it. Various things will need to be paid upfront and those expense need to come out of the earmarked funds. I could track that with a new account for the savings, but since I’m not setting up a new bank account the balances wouldn’t match up and I’d spend a lot of time transferring money in and out of the holiday account.

So I’ve introduced ‘Earmark’ as a new type of entity. In many ways they’re similar to accounts, but a single transaction can have both an account and an earmark, affecting the balance of both. So the transaction recording paying for our flights has the credit card as it’s account and the earmark set to the Germany holiday. It acts as an expense reducing the balance of both the account and the earmark.

Similarly, my wife’s income goes into our bank account and for this year I’m also earmarking it for Germany so it increases the bank balance (we did actually get the cash) and also increases the balance of the earmark.

This sounds like double counting, but the earmark balance is not included in the net worth which is still just the sum of account balances. The UI also displays “available funds” which is the sum of account balances less the sum of earmark balances.

The result of this is that an expense with an account and earmark reduces the balance of the account (we now have less money in total) but also reduces the balance of the earmark so there is less money set aside for that purpose. As a result the net worth is reduced but the available funds remains the same because the cost came out of our reserved funds.

Income with an account and an earmark increases the balance of both, so net worth goes up but available funds stays the same – those funds have been earmarked for that purpose and aren’t available for general use.

But what if we want to earmark a fixed amount per month separately to any actual income? To support that, Moolah allows creating transactions with an earmark but no account. In that case we’re recording the allocation of funds but with no actual cash movement.

The feature set is rounded out by tracking how much has been saved (total of income) and how much has been spent (total of expenses), a savings target and target date and providing some visualisations to show savings progress vs time and expense breakdowns.