Living in a state of accord.

Into Eth 2 – Adding Artemis

Continuing our adventures in setting up a private beacon chain… Previously we got an Eth 1 chain with the deposit contract and successfully sent a deposit to register a new validator.  So far though, we have no way of telling if it actually worked. What we need is an actual beacon chain client.  Enter Artemis…

Step 3 – Add a Beacon Chain Client

We’ll be using Artemis, partly because I’m incredibly biased and partly because it happens to support the exact version of the deposit contract we’ve deployed (what a coincidence!).

We’ll need a config file for it, the vast majority of which is boiler plate I copied and probably don’t actually need.  The important bit though is we need to point it at our Pantheon node so it can query the ETH1 chain data. We also need to tell it the address of our deposit contract:

# normal, test, simulation
# "test" pre-production
# "simulation" to run a simulation of deposits with ganache-cli, if a inputFile is included the file will replay the deposits
# "normal" production, must include contractAddr and nodeUrl
mode = "normal"
contractAddr = "0xdddddddddddddddddddddddddddddddddddddddd"
nodeUrl = "http://eth1:8545"

Note that “eth1” is the name we gave to our Pantheon docker image. Docker will do its magic to resolve that to Pantheon’s IP address inside the container.

And then we have some relatively standard docker boiler plate in our script for it.

If you’ve followed all the steps from part 1 and actually sent a deposit transaction, when you run Artemis after a few moments it should create a JSON file in artemis/output/artemis.json containing something like:

"amount": 32000000000,
"eventType": "Deposit",
"merkle_tree_index": "0",
"pubkey": "0x88526BB3800ABB7BA3E4DF4255D043AA661EDDBBFDC0B8C3209034B7EEA65EB3C32AAF0E5D19E8998A1CE5E2B9B64299",
"withdrawal_credentials": "0x008098BD37974616EC6DA256B5D2650E2363D011B7CEF902F7CCBFE938CBA0EA"

Which is a record indicating it has received and recognised our deposit. Pretty much nothing else happens though because we still don’t have enough validators to actually get the beacon chain started.  I suspect running some 65,000 validators on my laptop might be a little ambitious so the next step is likely to be tweaking config to reduce the number of validators required before the chain starts.

Into Eth 2 – Eth 1 and the Deposit Contract

I’ve started a little side-project to setup a “private beacon chain”.  The aim is to better understand how the beacon chain works and start to discover some of the things still required to be built or fixed in clients before it can officially launch.

So what is a private beacon chain? It’s intended to be an entirely self-contained, runs-on-my-laptop instance of the beacon chain, run as a small-scale simulation of how the real beacon chain will be fired up.

I’ve collected all the various scripts and config together in a git repo so you follow along from home if you like. Be warned, there is absolutely no polish and not even a lot of thought about organisation in there.

Step 1 – An Ethereum 1 Chain

First of all we need an Ethereum 1 chain that will be the source of our deposits for beacon chain validators. That’s pretty easy with Pantheon, but let’s complicate matters a little by using Docker. First we’re going to need a network which will eventually (hopefully) hold a whole bevy of different Eth 2 clients all happily interoperating on our beacon chain.  So:

docker network create beacontest

Then we need a genesis config for our private Eth 1 chain.  I’ve copied the genesis Pantheon uses by default in dev mode with a couple of tweaks I’ll explain later.  Here’s the full file.  There are a few accounts that have been allocated lots of ETH and the private keys are included in the config so it’s easy to import to MetaMask or other tools (don’t use these keys for anything real!).

Finally there’s a simple little script to run Pantheon using that file, in our docker network, exposing a bunch of ports and generally being setup to be useful for what we want to do.

Step 2 – Deposit Funds

Our beacon chain can’t become active until we have enough people who have deposited funds into the beacon chain contract to become validators. Obviously, that means we’ll need a beacon chain contract.

Step 2.1 Deploy the Deposit Contract

It turns out, the beacon chain contract has changed a bunch of times in different versions of the spec. This is painfully confusing because docs or examples you read usually don’t mention which version of the spec and you can waste many hours trying to interact with the wrong contract.

We will be using version 0.7.1 of the spec. Here’s that version of the deposit contract spec and the deposit contract code.

To make life simpler, I’ve taken the runtime byte code for that contract and added it into the genesis config at the fixed address 0xdddddddddddddddddddddddddddddddddddddddd. Note when including contracts in the genesis config you need the runtime byte code, not the constructor byte code you’d normally send as transaction data. Remix makes this easy with it’s “Runtime Bytecode” tab.

Step 2.2 Deposit Some ETH

The deposit contract spec helpfully tells us that there’s a deposit method we need to call which takes three arguments: pubkey: bytes[48], withdrawal_credentials: bytes[32], signature: bytes[96]

Unhelpfully it gives us almost no clue what those parameters actually mean or how to generate them.

I haven’t yet found a nice stand-alone way to generate the required parameters, so I wound up shoving an extra class into my local checkout of the Artemis codebase – right next to the existing which has the main method.  There are three key steps:

  1. Generate two BLS key pairs. One for the validator to use and one for the ETH 2 account that our funds will be sent to if we choose to leave the validator pool.
    • The withdrawal keys can be kept offline until you actually withdraw the funds whereas the validator keys need to be online so the validator can sign things and do its job.
  2. Encode the public key of the validator key pair using the compressed form with big endian encoding.
    • This will be the first argument – pubKey.
    • Don’t ask me why it’s called compressed encoding when it appears to be exactly the same length as the uncompressed form. UPDATE: Ben Edgington helpfully points out that an uncompressed key is actually 96 bytes – I’d been misled by a well-meaning error message when the real problem was I was using little endian instead of big.
  3. Calculate the SHA256 hash of the public key of the withdrawal key pair, in big endian encoding. Replace the first byte with 0 (the BLS_WITHDRAWAL_PREFIX_BYTE).
    • This will be the withdrawal commitment
  4. Calculate the signature. This requires serialising the DepositData object that will ultimately be created by Eth2 clients in response to our deposit in SSZ with hash trees and stuff. Don’t ask me, I just called the existing Artemis code… That then gets signed using the validator’s private key.
    • This proves you actually have the private key matching the public key you’re trying to register as a validator.
    • WARNING: This signature is not checked by the deposit contract, only be Eth2 clients. So if you get it or the public key wrong your deposit will be ignored by the beacon chain and you won’t get your ETH back.

Now that we know what parameters we need to pass, we just need to create a transaction with the required 32 ETH and those parameters encoded as a call to the deposit method.  I wrote an especially ugly bit of JavaScript to do that. You’ll need to install the npm dependencies first with:

npm install

and can then send our deposit with:

node index.js

At this point, if we actually had an ETH2 client, it would recognise that deposit as creating our first validator. Setting up our ETH2 client will be step 3 but I’m not writing that up today.  And, spoiler alert, we’re actually going to need another 65535 validators before the beacon chain will actually start, but that’s a problem for another day.

Ethereum State Rent Proof of Concept

I’ve had the opportunity to do some proof of concept development of the Ethereum state-rent proposal that Alexey Akhunov has been leading on the Pantheon code base. The proposal evolved as the work continued so the actual implementation is now a lot simpler than described in that PDF.

Note that the aim is to explore what it takes to implement the proposal, not to create production ready code.  The current work is all available on on my state-rent branch.

The easiest way to begin exploring the changes required to implement state-rent in Pantheon is to start with the hard forks required which translate to new ProtocolSpecs in Pantheon which are defined in MainnetProtocolSpecs. Ultimately there should be four hard forks:

  1. Introduce replay protection for accounts that are evicted
  2. Charge fixed state rent for “owned” accounts
  3. Charge state rent for newly allocated storage
  4. Charge state rent for pre-existing storage

Fork 1: Replay Protection

When accounts are evicted, their nonce would reset to 0, potentially enabled old transactions to be replayed. There are a few ways to prevent this, my preference was to introduce temporal production (variant 2 in the PDF) which is a simple extra validation to perform before processing transactions.  As such, I skipped implementing it in the proof of concept.

Fork 2: State Rent for Owned Accounts

In this milestone “owned” accounts are charged a fixed rent per block. An “owned” account is defined as one without code in the PoC. To avoid having to process every account on every block, the rent is only recalculated when the account is changed at the end of the block (ie a change would already have to be written to disk for it).

TODO: Formalise the definition of owned account more. Particularly, clarify if an account created with empty code is considered owned or not.

The ProtocolSpec for this fork is defined based on Constantinople with three changes:

public static ProtocolSpecBuilder<Void> stateRentOwnedAccountsDefinition(
      final int chainId, final long rentEnabledBlockNumber) {
    return constantinopleDefinition(chainId)
            rentCost -> new OwnedAccountsStateRentProcessor(rentCost, rentEnabledBlockNumber))

Firstly, we set a rent cost of 2 Gwei/block. This is a pretty arbitrary figure and will likely need tweaking. This value is used by the rent processor we also set but has been split out so it’s easy for a hypothetical future fork to change the rent cost without having to implement a completely different rent processor.

Finally, we set a StateRentOwnedAccountsAccountInit which is used to setup defaults for any newly created accounts (post-fork). In this case, it sets the rent balance to zero and rent block to the current block number. These are two new fields added to account state in this fork. This is the first time the new account process has varied in a hard fork, so there was a bunch of plumbing code required to add the concept of a pluggable AccountInit but none of it difficult.

However, it’s also the first hard fork that adds fields to account state, which makes serialising and deserialising them a little more difficult.  That code is in AccountSerializer. Fortunately it’s easy in RLP to check if you are at the end of the current list or not, so when deserialising we can tell if rent balance and rent block have been added to this account state by checking if we’re at the end of the list (line 51).

Calculating the rent owed is handled by OwnedAccountsStateRentProcessor with much of the code able to be reused with later forks via AbstractRentProcessor. The rent balance is one of the very few things in Ethereum that can be negative which makes life a little frustrating as the existing Wei class used for account balance can’t be reused and there’s a few points where things need to be converted between Wei and BigInteger but it’s manageable.

Rent is only recalculated when the account has been changed in the block. In Pantheon the simplest way to achieve this was to piggy-back on the concept of “touched” accounts used in Spurious Dragon.  We simply iterate through the touched accounts and check if the root of their account state trie is dirty (which is also already tracked), if it is we recalculate rent via the RentProcessor.

At this stage, any account which is unable to pay its rent can be deleted in the same way it would have been if it were empty. Again, simple to reuse existing code for this.

For the record, actually calling the rent processor is hooked into MainnetBlockProcessor so that it happens as the last step in processing a block.

TODO: Additional clarification is required about exactly when rent should be charged. In the PoC it’s applied even after paying the miner which seems sensible but there are a number of things happening at the end of a block so it needs to be completely clear.

Note that the priority queue for eviction has been scrapped (I forget why it was originally required…).

Fork 3: State Rent for New Storage

This fork introduces quite a bit of new stuff, but most importantly adds a new storagesize field to account state which is adjusted as part of every SSTORE call. Existing storage is not included in the storagesize field at this point, only a count of the storage size either added or cleared since fork 3 became active. The field is added when new accounts are created, when rent is recalculated for an existing account or when an SSTORE call results in the account storage size changing.

TODO: Need to be clear about exactly when this field is first added (particularly if multiple SSTORE calls are made that leave the storage at the same size or even completely unchanged).

The ProtocolSpec for this fork is based on the one for fork 2 but with a few changes:

  public static ProtocolSpecBuilder<Void> stateRentNewStorageDefinition(
      final int chainId, final long rentEnabledBlockNumber) {
    return stateRentOwnedAccountsDefinition(chainId, rentEnabledBlockNumber)
            rentCost -> new StorageSizeStateRentProcessor(rentCost, rentEnabledBlockNumber))

There’s a new AccountInit to set storagesize to 0 when a new account is created.

Rent cost is reduced to 1 Gwei since it’s now charged per byte per block. This is almost certainly still the wrong value but it definitely should be less than when rent was purely per block.

We also supply a new RentProcessor, StorageSizeStateRentProcessor, which applies to all accounts whether they have code or not. It also charges rent based on the value of the new storagesize field.  If storagesize hasn’t yet been set, it’s set to a fixed value to represent the size of an empty account (73 which is the length of an empty account RLP in bytes).

TODO: The PoC deliberately uses the value of storagesize from the start of the block, ignoring any changes applied in the current block. Otherwise, if an account is untouched for 100 blocks, then increases its storage size, it will be charged rent at the increased storage size for the whole 100 blocks it was untouched.  This rule may need to be tweaked slightly to use the new storagesize value for the current block and the old one for the untouched blocks but it seems reasonable to defer rent changes until the block after they take effect – increased storage would have cost gas in the current block and reduced storage doesn’t really give a benefit until the block is complete and committed anyway.

NOTE: storagesize may be negative during this fork, indicating that the account now uses less storage than it did when this fork first happened.  After fork 5, storagesize should only ever be positive. 

TODO: Rent calculation during this fork should probably use max(storagesize, 0) to avoid the rent due being negative.

Accounts with code are not deleted when they can’t pay their rent, they are instead “evicted” and replaced with a hash stub so they can be restored via the new RESTORETO operation (see below). This is one of the more complicated changes to work through the code – firstly just in terms of tracking evicted accounts through the transaction commit/rollback code but also in terms of detecting them in the account state trie.

Hash Stubs

Challenging bit: Determining if an address in the account state trie is an actual account state or just a hash stub.

Looking again at AccountSerializer in the serializeHashStub method, we can see that a hash stub is an RLP list containing the code hash and storage root hash. RLP doesn’t provide any typing information though so to detect a hash stub, we check if the first item is a nonce or a hash. If it is a hash stub, the size of the RLP item must be 32 bytes. Theoretically a nonce is an arbitrary unsigned scalar number, so it could wind up being 32 bytes (a 256bit number) but at least in Pantheon we already don’t support nonces that big and you’d have to spend a lot of eth on gas to get a nonce to make a nonce that big. So the item length appears to be a suitable way to distinguish between account state and hash stubs.

UPDATE: A smarter way to do this would have been to just count the number of items in the RLP list which is relatively straight-forward. If there are 2 items, it must be a hash stub.

TODO: Clarify interactions for any operation that looks up an account state if the account state has been replaced by a hash stub. Currently Pantheon treats hash stubs as if they don’t exist except in the RESTORETO operation. This is mostly right, except that contract creation should consider it an address conflict if a hash stub is at the target address (not currently implemented but pretty straight forward).

New EVM Operations

There are three EVM operations added in this fork: PAYRENT, RENTBALANCE, SSIZE and RESTORETO. Additionally SSTORE is updated to update the storagesize field. PAYRENT, RENTBALANCE and the SSTORE changes are pretty trivial but RESTORETO has quite a few complexities.

TODO: Decide on the gas cost for these new opcodes.

Tool Support: RENTBALANCE returns an Int256 (signed) which I think is the only time an EVM operation returns a signed number. Need to double check compatibility for this with things like Solidity but should be fine.

Restore To Operation

The RESTORETO operation turns out to be one of the more interesting parts of this spec to implement.  The basic workflow is slightly nuanced and involves three accounts.  Let’s say that Account A is evicted, leaving behind a stub with its code and storage root hashes.  To restore Account A:

  1. Account B is created with the same code as Account A.
  2. Account C is created with code that sets up the storage in Account C to precisely match the storage Account A had before being evicted. It may do this in it’s constructor or expose functions that can be called over time to build up the correct state from multiple sources (potentially sharing the gas cost of rebuilding that state across multiple people).
  3. Account C calls RESTORETO <address of Account A> <address of Account B>.  The EVM checks that account B’s code hash matches what Account A’s stub records, and that Account C’s storage root hash matches what Account A’s stub records.  If so, it creates a new contract at Address A, replacing the hash stub, with the code from Account B and the storage from Account C. It also transfers any balance from Account C to Account A and destroys Account C (just as SELFDESTRUCT would).

Mostly this is fairly straight-forward to implement as seen in The hidden catch is that most Ethereum clients, including Pantheon, don’t update the storage trie until the transaction actually commits – the changes made mid-transaction are stored in a simple HashMap which is much faster. As such, getting the storage root hash for Account C isn’t straight-forward. In fact, in the PoC I haven’t implemented it at all. It’s certainly possible to create a copy of the unmodified Trie, update that with pending changes and calculate the storage root but it’s an expensive operation and that code would only be exercised by RESTORETO, significantly increasing the testing required to fully cover RESTORETO.

TODO: Find a way to change RESTORETO so that the storage root is more easily available when required.

One option here would be to have execution immediately halt when RESTORETO is called, and then to perform the actual restore at the end of the transaction, much like how SELFDESTRUCT only applies at the end of the transaction.  Geth and Pantheon currently update the storage trie after every transaction (they don’t necessarily calculate the root hash but that’s simple once you have the trie), but TurboGeth delays updating the trie until the end of the block, improving performance if the same account is used from multiple transactions in one block.  Implementing that improvement for Pantheon is also planned.  Delaying application of RESTORETO until the end of the block feels a bit too weird though.

More thought is required in this area to find a solution that both makes sense to users and can be implemented without significantly increasing the surface area required for testing.

Fork 4: State Rent for Existing Storage

The PoC doesn’t include any of this fork currently, but it’s fairly straight-forward. When an account’s rentblock field is first updated to a block after fork 4 is in effect, the account’s storagesize field is increased by the size of its storage immediately prior to fork 3 coming into effect. Basically, we add the historical state size in.

This is done as a separate fork because it gives times for client developers to precompute the storage size for every account on the various public networks at the fork 3 block and include that in a client update. This means the cost of calculating that existing storage size is done offline and doesn’t impact on performance of every node.

However, because the storagesize field is only updated when the account is first touched anyway, it is still feasible to calculate the existing storage size on the fly, either because the precomputed values aren’t available for that particular network or in order to verify them.

TODO: It’s not explicitly stated anywhere at the moment, but when calculating the rent due for the first time after fork 4, rent should be calculated up to fork 4 block using the new-storage-only-value, then add in the existing storage size and calculate rent from fork 4 block up to the current block. Otherwise accounts start being charge rent for existing storage at different times which is too hard to understand (and unfair).

Other Work Required

There’s some other stuff that would need to be done to make state-rent production ready:

  • Additional JSON-RPC methods such as getRentBalance and probably something to calculate the rent owing for a given account at a specific block.
  • CALLFEE opcode. The spec refers to this to provide a way for contracts to require a rent contribution when users call into it, but there weren’t enough details to actually implement it. The main question is where the extra fee is taken from (it can’t be an extra gas charge because rent is tracked in ETH not gas).
  • Currently if you deploy a contract with Remix it doesn’t get any eth balance so it is immediately evicted when you interact with it. This is quite confusing and frustrating as a user.  We’ll either need to update tools to allow sending an initial rent balance, provide some free rent on contract creation or allow a grace period after contract creation before rent is charged.

Introducing Pantheon

This week, the work I’ve been doing for the past 6 months, and that PegaSys has been working on for the past 18 months or so was released into the world. Specifically we’ve released Pantheon 0.8.1, our new MainNet compatible, Apache 2 licensed, Java-based Ethereum client. And it’s open source.

I’m pretty excited about it on a few fronts.  Firstly I think it’s a pretty important thing for the Ethereum community. To be a healthy ecosystem, Ethereum needs to have diversity in its clients to avoid a bug in one client taking out or accidentally hard forking the entire network. Currently though, Geth and Parity dominate the Ethereum client landscape.  Pantheon clearly won’t change that in the short term, but it is backed by significant engineering resources to help it keep up with the ever changing Ethereum landscape and be a dependable option.

I’m also really excited that Pantheon is released under the Apache 2.0 license. Both Parity and Geth along with most other clients are licensed under the GPL or LGPL. There are still a large number of enterprises that completely avoid the GPL and LGPL which has closed off Ethereum to them. Having Pantheon available under a permissive license and in a highly familiar language like Java will make it much easier for many enterprises to start using, building on and innovating with Ethereum.

Pantheon will also be building out functionality from the Enterprise Ethereum standard for things like privacy and permissioning to make private chains more powerful and flexible. Meanwhile we have a significant number of researches continuing to work on developing new ways to get the most out of Ethereum.

Finally I’m quite excited to be able to contribute to an open source project as my full time job. I’ve had the opportunity to do some open source for work in the past but only on a fairly small scale. It’s a bit daunting to have everything in the open and on the record, but I’m really looking forward to engaging with the community and being able to show exactly what I’ve been doing.