Symphonious

Living in a state of accord.

Exploring Ethereum: Ommers vs Non-Canonical Blocks

One subtle detail in the way Ethereum works is that there is a difference between Ommers and Non-Canonical Blocks. It’s common for people to use the term Ommer for both of these and most of the time the difference doesn’t matter but sometimes it does.

So what is a non-canonical block? Non-canonical blocks are ones which a client imports but which don’t wind up on the canonical chain. Maybe they were on the canonical chain for a while and then a re-org switched to a different chain or maybe they were imported to a fork and spent their entire life languishing there. Either way, they don’t form part of the current consensus chain and have absolutely no effect on the world state. It’s like their transactions have never been executed and no one gets any form of miner reward for them. Non-canonical blocks must be entirely valid blocks that could form part of the canonical chain, but weren’t because we found a better chain.

Apart from not getting any mining rewards, that probably sounds a lot like an Ommer, and it is. But actually an Ommer is just a block header which has been included in the Ommer list of another block. The block does not have to be completely valid – it just has to have a valid proof of work solution and be from a fork that started within the last 6 blocks of the block that included it. The transactions are not included so there’s no verification that they were in any way valid. The miner of the Ommer gets a small reward when it is included as an Ommer and the miner of the block that includes it also gets an award for doing so.

A block can be both an Ommer and a non-canonical block, and in practice usually are. However, it is possible for a block that would be invalid to import as a canonical or non-canonical block to be included as an Ommer.

Moolah Diaries: Making inject-loader and vuetify-loader play nice

I’ve been upgrading Moolah after a long period of neglect. One of the tasks is to update it to WebPack 4. Mostly this went smoothly by just creating a new project with vue-cli and bringing its generated build setup over to replace the old one.  Then a bunch of tweaking.

One thing that did bite however, was tests using inject-loader started failing as soon as I added vuetify-loader to the project with:

Error: Module parse failed: 'import' and 'export' may only appear at the top level

There may be a better way to fix this in config, but the simple answer is to adjust the inject-loader import line from something like:

import categoryStoreLoader from 'inject-loader!categoryStore';

to:

import categoryStoreLoader from 'inject-loader!babel-loader!categoryStore';

Adding in the extra babel-loader ensures the imports are processed and replaced by require and everything works.

 

Into Eth 2 – Adding Artemis

Continuing our adventures in setting up a private beacon chain… Previously we got an Eth 1 chain with the deposit contract and successfully sent a deposit to register a new validator.  So far though, we have no way of telling if it actually worked. What we need is an actual beacon chain client.  Enter Artemis…

Step 3 – Add a Beacon Chain Client

We’ll be using Artemis, partly because I’m incredibly biased and partly because it happens to support the exact version of the deposit contract we’ve deployed (what a coincidence!).

We’ll need a config file for it, the vast majority of which is boiler plate I copied and probably don’t actually need.  The important bit though is we need to point it at our Pantheon node so it can query the ETH1 chain data. We also need to tell it the address of our deposit contract:

[deposit]
# normal, test, simulation
# "test" pre-production
# "simulation" to run a simulation of deposits with ganache-cli, if a inputFile is included the file will replay the deposits
# "normal" production, must include contractAddr and nodeUrl
mode = "normal"
...
contractAddr = "0xdddddddddddddddddddddddddddddddddddddddd"
nodeUrl = "http://eth1:8545"

Note that “eth1” is the name we gave to our Pantheon docker image. Docker will do its magic to resolve that to Pantheon’s IP address inside the container.

And then we have some relatively standard docker boiler plate in our run.sh script for it.

If you’ve followed all the steps from part 1 and actually sent a deposit transaction, when you run Artemis after a few moments it should create a JSON file in artemis/output/artemis.json containing something like:

{
"amount": 32000000000,
"eventType": "Deposit",
"merkle_tree_index": "0",
"pubkey": "0x88526BB3800ABB7BA3E4DF4255D043AA661EDDBBFDC0B8C3209034B7EEA65EB3C32AAF0E5D19E8998A1CE5E2B9B64299",
"withdrawal_credentials": "0x008098BD37974616EC6DA256B5D2650E2363D011B7CEF902F7CCBFE938CBA0EA"
}

Which is a record indicating it has received and recognised our deposit. Pretty much nothing else happens though because we still don’t have enough validators to actually get the beacon chain started.  I suspect running some 65,000 validators on my laptop might be a little ambitious so the next step is likely to be tweaking config to reduce the number of validators required before the chain starts.

Into Eth 2 – Eth 1 and the Deposit Contract

I’ve started a little side-project to setup a “private beacon chain”.  The aim is to better understand how the beacon chain works and start to discover some of the things still required to be built or fixed in clients before it can officially launch.

So what is a private beacon chain? It’s intended to be an entirely self-contained, runs-on-my-laptop instance of the beacon chain, run as a small-scale simulation of how the real beacon chain will be fired up.

I’ve collected all the various scripts and config together in a git repo so you follow along from home if you like. Be warned, there is absolutely no polish and not even a lot of thought about organisation in there.

Step 1 – An Ethereum 1 Chain

First of all we need an Ethereum 1 chain that will be the source of our deposits for beacon chain validators. That’s pretty easy with Pantheon, but let’s complicate matters a little by using Docker. First we’re going to need a network which will eventually (hopefully) hold a whole bevy of different Eth 2 clients all happily interoperating on our beacon chain.  So:

docker network create beacontest

Then we need a genesis config for our private Eth 1 chain.  I’ve copied the genesis Pantheon uses by default in dev mode with a couple of tweaks I’ll explain later.  Here’s the full file.  There are a few accounts that have been allocated lots of ETH and the private keys are included in the config so it’s easy to import to MetaMask or other tools (don’t use these keys for anything real!).

Finally there’s a simple little script to run Pantheon using that file, in our docker network, exposing a bunch of ports and generally being setup to be useful for what we want to do.

Step 2 – Deposit Funds

Our beacon chain can’t become active until we have enough people who have deposited funds into the beacon chain contract to become validators. Obviously, that means we’ll need a beacon chain contract.

Step 2.1 Deploy the Deposit Contract

It turns out, the beacon chain contract has changed a bunch of times in different versions of the spec. This is painfully confusing because docs or examples you read usually don’t mention which version of the spec and you can waste many hours trying to interact with the wrong contract.

We will be using version 0.7.1 of the spec. Here’s that version of the deposit contract spec and the deposit contract code.

To make life simpler, I’ve taken the runtime byte code for that contract and added it into the genesis config at the fixed address 0xdddddddddddddddddddddddddddddddddddddddd. Note when including contracts in the genesis config you need the runtime byte code, not the constructor byte code you’d normally send as transaction data. Remix makes this easy with it’s “Runtime Bytecode” tab.

Step 2.2 Deposit Some ETH

The deposit contract spec helpfully tells us that there’s a deposit method we need to call which takes three arguments: pubkey: bytes[48], withdrawal_credentials: bytes[32], signature: bytes[96]

Unhelpfully it gives us almost no clue what those parameters actually mean or how to generate them.

I haven’t yet found a nice stand-alone way to generate the required parameters, so I wound up shoving an extra class into my local checkout of the Artemis codebase – GenerateKeys.java right next to the existing Artemis.java which has the main method.  There are three key steps:

  1. Generate two BLS key pairs. One for the validator to use and one for the ETH 2 account that our funds will be sent to if we choose to leave the validator pool.
    • The withdrawal keys can be kept offline until you actually withdraw the funds whereas the validator keys need to be online so the validator can sign things and do its job.
  2. Encode the public key of the validator key pair using the compressed form with big endian encoding.
    • This will be the first argument – pubKey.
    • Don’t ask me why it’s called compressed encoding when it appears to be exactly the same length as the uncompressed form. UPDATE: Ben Edgington helpfully points out that an uncompressed key is actually 96 bytes – I’d been misled by a well-meaning error message when the real problem was I was using little endian instead of big.
  3. Calculate the SHA256 hash of the public key of the withdrawal key pair, in big endian encoding. Replace the first byte with 0 (the BLS_WITHDRAWAL_PREFIX_BYTE).
    • This will be the withdrawal commitment
  4. Calculate the signature. This requires serialising the DepositData object that will ultimately be created by Eth2 clients in response to our deposit in SSZ with hash trees and stuff. Don’t ask me, I just called the existing Artemis code… That then gets signed using the validator’s private key.
    • This proves you actually have the private key matching the public key you’re trying to register as a validator.
    • WARNING: This signature is not checked by the deposit contract, only be Eth2 clients. So if you get it or the public key wrong your deposit will be ignored by the beacon chain and you won’t get your ETH back.

Now that we know what parameters we need to pass, we just need to create a transaction with the required 32 ETH and those parameters encoded as a call to the deposit method.  I wrote an especially ugly bit of JavaScript to do that. You’ll need to install the npm dependencies first with:

npm install

and can then send our deposit with:

node index.js

At this point, if we actually had an ETH2 client, it would recognise that deposit as creating our first validator. Setting up our ETH2 client will be step 3 but I’m not writing that up today.  And, spoiler alert, we’re actually going to need another 65535 validators before the beacon chain will actually start, but that’s a problem for another day.

Adding a DCO Signed-off-by to every commit in a git repo

If you’re switching from a CLA model to a DCO model you may want to add a Signed-off-by line to every existing commit so that you can run automated DCO checks over the entire repository.  Doing this obviously assumes that your CLA sets things up so that you are actually in a position to provide a DCO sign-off.

Once you’re happy with all the legalities, the change is a single command using git filter-branch:

git filter-branch --msg-filter "cat - && echo && echo 'Signed-off-by: Some Person <some.person@example.com>'"