Symphonious

Living in a state of accord.

Hard Truths for ETH Stakers

If you’re getting into staking ETH on the beacon chain, it’s important to know what you’re actually signing up for. There’s lots of articles about how easy it is and how awesome, intelligent and good looking stakers are, so here’s a few hard truths you should make sure you’re ok with…

Short Term Stuff

Most people know this stuff so I’m just going to skim over it.

No Withdrawals Yet

You can’t withdraw staked ETH or rewards yet. There isn’t a fixed timeline on when you will be able to.

Penalties

Rewards on the beacon chain are not automatic – you actually have to run a node and perform the assigned duties.  If you fail to perform the duties, instead of a reward you’ll actually get a small penalty.

If lots of validators aren’t performing their duties well at the same time and the chain stops finalising, those small rewards will gradually get bigger and bigger.

Slashings

If your validator signs something that’s self-conflicting (ie not just wrong but disagreeing with what you’ve previously said) it can be slashed. This is a bigger penalty and your validator is immediately deactivated so you can’t earn any more rewards (and still can’t withdraw). How big depends on how many other validators are slashed at the same time – it can be anything from 0.5ETH to your entire stake).

Rewards  Automatically Reduce as Validators Activate

As more validators begin staking, the rewards paid are automatically reduced. So while you might have earned 0.00004 ETH for a correct attestation last week, this week it may be only 0.00003 ETH, and next week it may be even less.

Which brings us to the first key principal you should be ok with…

Key Principals

Pay the Minimum Amount Possible to Secure the Chain

The philosophy of Ethereum is to pay the minimum amount possible to ensure the chain remains secure.  This applies equally to miners in Eth1 and stakers in Eth2. That’s why rewards automatically reduce when there are more active validators – the chain has enough validators to be secure so it doesn’t need to provide as much incentive for staking. But it goes well beyond that.

The algorithm for determining those rewards may itself be changed to further reduce them if in the future it’s deemed that security would still be sufficient. There is absolutely no guarantee of any rate of reward.

In fact, there is already one proposal that could reduce rewards which was considered for the first protocol upgrade but ultimately seen as unnecessary. Specifically the proposal was to cap the number of validators that can actually be active at any one time, with the rest being dormant (so not earning any rewards) and randomly cycle which validators are dormant. It was seen as unnecessary because even if we continued activating validators at the maximum possible rate, it wouldn’t hit the cap and activate these changes for at least another year. At some point though it’s likely to be implemented as it provides important reduction in the load required for nodes following the chain.

Which brings us to…

Ethereum Puts Itself First

Whenever there is a trade off being individuals or groups and the safety, security and reliability of the Ethereum chain, the Ethereum protocol and Ethereum developers will always prioritise the chain first. After all, if the chain isn’t working no one can benefit from it.

Some examples of this in the beacon chain are the long delays before deposits activate – the chain is ensuring the validator set doesn’t change too fast because that’s a security risk. The fact that you have to wait 2+ weeks for your validator to activate is just unfortunate for you.

Note though, that this isn’t intentionally capricious. If the chain would function just as well either way, then sure, make the protocol as nice to people in any role as possible. This also ties into paying the minimum amount for security – if we can make the staking experience better in some way (that doesn’t reduce security) then people will likely be prepared to stake for less money and rewards can be reduced.

Ethereum Does Not Exist To Make Stakers Rich

This is really just an extension of the previous two, but it’s worth being extra clear. Staking is fun for the whole family and everything but it really is just a means to an end – providing security for the Ethereum chain. You can lose sight of that because at the moment the beacon chain’s only function is staking but that’s just a temporary thing. Ultimately Eth1 will merge in and all that staking will just be to provide security for contracts and transactions on the chain.

Remember, stakers are service providers to the network and they’re paid for their services with rewards.

Different people have different views on exactly what the point of Ethereum is (world computer, digital asset, money, all of the above?) but it’s definitely not to make stakers rich. Don’t expect any sympathy if it winds up being hard to make a profit – unless security is at risk, rewards won’t be increased (“Pay the Minimum Amount Possible to Secure the Chain” remember?).

Change Is Inevitable

Ethereum is a blockchain that has always and will continue to change and upgrade. There are an unusually large number of big changes coming as the collection of technologies referred to as Eth2 roll out, but it won’t ever stop. There will always be new ideas to take advantage of and new problems to solve.

Those changes will be (or at least are intended to be) great for Ethereum overall, but they might not be for any particular person or group. They might mean stakers need more resources to keep up (e.g. the merge requiring running a full Eth1 node), they might affect reward rates or any number of other things. Just because something is true today, doesn’t mean it always will be.

The Good News

That all sounds pretty doom and gloom, but the good news is that staking is entirely optional. There are plenty of other places you can put your money if you don’t like the deal that staking is offering.

Most importantly, once withdrawals are possible, it’s also really easy to stop staking and get your stake back (assuming you didn’t go getting yourself slashed badly…). It’s like if you could buy a mining rig with a guarantee that it could be sold for the original price when you’re done with it and you get to keep whatever profit or loss you’ve earned.

 

PS: I’ve been informed that staking also doesn’t guarantee to make you awesome, intelligent or good looking.

Exploring Eth2: Attestation Rewards and Validator Performance

Through the beacon chain testnets people have been gradually developing better ways to evaluate how well their validators are working. Jim McDonald made a big leap forward defining attestation effectiveness which focuses on the inclusion distance for attestations. That revealed a bunch of opportunities for clients to improve the attestation inclusion distance and attestation effectiveness scores improved quite dramatically until now scores are generally at or close to 100% pretty consistently.

But then beaconcha.in added a view of how much was earned for each attestation. Suddenly people were left wondering why some of their “rewards” were negative even though the attestation was included quickly. It became obvious that inclusion distance was only one part of attestation rewards.  

How Attestation Rewards Work

There are four parts in total:

  • Reward based on inclusion distance
  • Reward or penalty for getting the source correct
  • Reward or penalty for getting the target correct
  • Reward or penalty for getting the head correct

It turns out that inclusion distance is the smaller of these components.  Ben Edgington explains all the detail very well in his annotated spec. First we calculate the base reward, which is the component which factors in the validator’s effective balance (lower rewards if your effective balance is less than the maximum 32ETH) and the total staked ETH (reducing rewards as the number of validators increase).

Then for the source, target and head attestations, a reward or penalty is applied depending on whether the attestation gets them right.  A missed attestation is penalised for all three. The penalty is 1 * base reward. The reward however factors in a protection against discouragement attacks so is actually base reward * percentage of eth that attested correctly. This provides incentive for your validator to do whatever it can, like relaying gossip well, to help everyone stay in sync well.

Finally the inclusion distance reward is added. There’s no penalty associated with this, missed attestations just don’t get the reward. 1/8th of this reward is actually given to the block proposer as reward for including the attestation, so the maximum available reward for the attester is 7/8th of the base reward. So it winds up being (7/8 * base reward) / inclusion distance.

So inclusion distance is actually the smallest component of the reward – not only is it only 7/8th of base reward, there’s no penalty attached. The most important is actually getting source right as attestations with incorrect source can’t be included at all, resulting in a 3*base reward penalty. Fortunately getting source right is also the easiest because it’s back at the justified checkpoint so quite hard to get wrong.

Evaluating Validators

If we’re trying to evaluate how well our validator is doing, these calculations add quite a lot of noise that makes it difficult. Especially if we want to know if our validator is doing better this week than it was last week.  If you just look at the change in validator balance, how many block proposals you were randomly assigned dominates and even just looking at rewards for attestations is affected by the number of validators and how well other people are doing.

I’d propose that we really want to measure what percentage of available awards were earned. That gives us a nice simple percentage and is highly comparable across different validators and time periods.

The first step is to think in terms of the number of base rewards earned which eliminates variance from the validator’s balance and total eth staked.  Then we can ignore the discouragement attack factor and say that each of source, target and head results in either plus or minus 1 base reward. Inclusion distance is up to 7/8th of a base reward, scaling with distance like normal. Like for attestation effectiveness, we’d ideally also use the optimal inclusion distance – only counting the distance from the first block after the attestation slot to avoid penalising the validator for empty slots they couldn’t control. In practice this doesn’t make a big difference on MainNet as there aren’t too many missed slots.

So each attestation duty can wind up scoring between -3 and +3.875 (3 and 7/8ths). For any missed attestation, the score is -3. For any included attestation, we calculate the 4 components of the score with:

Inclusion distance score: (0.875 / optimal_inclusion_distance)

Source score: 1 (must be correct)

Target and head scores: 1 if correct, -1 if incorrect

And to get a combined score, we need to add them together.

We can actually do this with the metrics available in Teku today:

(
validator_performance_correct_head_block_count - (validator_performance_expected_attestations - validator_performance_correct_head_block_count) +

validator_performance_correct_target_count - (validator_performance_expected_attestations - validator_performance_correct_target_count) +

validator_performance_included_attestations - (validator_performance_expected_attestations - validator_performance_included_attestations) +

(0.875 * validator_performance_included_attestations / validator_performance_inclusion_distance_average)
) / validator_performance_expected_attestations / 3.875

Which is essentially :

(
(correct_head_count - incorrect_head_count +
correct_target_count - incorrect_target_count +
included_attestation_count - missed_attestation_count
) / expected_attestation_count +
0.875 / inclusion_distance_average
) / 3.875

To give a score between 0 and 1 pretty closely approximating the percentage of possible attestation rewards that were earned. With the discouragement attack factor ignored, we are slightly overvaluing the source, target and head rewards but it’s very minimal. On MainNet currently they should be +0.99 when correct instead of +1 so doesn’t seem worth worrying about.

You can also calculate this fairly well for any validator using a chaind database with a very long and slow SQL query.

Evaluating the Results

Looking at the numbers for a bunch of validators on MainNet, generally validators are scoring in the high 90% range with scores over 99% being common but over anything more than a day it’s hard to find a validator with 100% which generally matches what we’d expect given the high participation rates of MainNet but knowing that some blocks do turn up late which will likely result in at least the head being wrong.

One key piece of randomness that’s still creeping into these scores though are that its significantly more likely to get the target wrong if you’re attesting to the first slot of the epoch – because then target and head are the same and those first blocks are tending to come quite late.  There are spec changes coming which should solve this but it is still affecting results at the moment.

What About Block Rewards?

For now I’m just looking at what percentage of blocks are successfully proposed when scheduled (should be 100%).  There are a lot of factors that affect the maximum reward available for blocks – while blocks aren’t reaching the limit on number of attestations that can be included, I’d expect that all the variation in rewards comes down to luck.  Definitely an area that could use some further research though.

Exploring Ethereum 2: Weak Subjectivity Period

Occasionally the term “weak subjectivity period” pops up in Eth2 discussions. It’s a weird concept that you can usually just watch fly by and not miss too much. But when you’re talking about how to sync an existing Eth2 chain it becomes quite important.  Probably the best resource for it is Vitalik’s post: Proof of Stake: How I Learned to Love Weak Subjectivity I’ve struggled to get my head around it and why it matters so am writing up my current understanding. There is almost certainly at least one mistake in here somewhere…

So what is the weak subjectivity period? It’s the period that a client can be offline for and when it comes back online be able to completely reliably process blocks to get to the consensus chain head. For proof of work you can always do this, but not for proof of stake. To see why not, let’s look at an example.

Say we have an Eth2 network chugging away. Once 2/3 of those validators have attested to a particular epoch it’s considered finalised and no re-orgs can change it. In order to finalise two conflicting epoch’s you’d need at least 1/3 of validators to sign conflicting attestations but doing so is a slashable offence so there’s a very strong economic incentive to not do that. That incentive is essentially what crypto-economics are all about, whether you’re talking PoW or PoS it’s not that it’s mathematically impossible to break the chain, but that it costs you more money than anyone is willing or able to spend.

At this point it sounds like you should be able to just process blocks reliably, confirm the attestations and signatures all line up and it all works out. What’s the catch?

The catch is that validators can withdraw their staked funds and stop being a validator. There are limits on how fast those withdrawals can happen but once the money is out the economic incentive to not misbehave is gone. Critically despite the validator having withdrawn the money, they still have their private key and can sign things – with no staked funds anymore they can’t be slashed for it. Nodes fully sync’d to the chain know they are no longer a validator and reject those signatures but nodes further behind don’t yet have that information and see the signature as valid.

So, if we have 1/3 of validators which have withdrawn their stake, if my node is far enough back on the chain to have not seen the withdrawal of any of those nodes, then 1/3 of the validators you currently think are valid have no incentive to be honest and can sign any blocks or attestations with complete impunity and potentially form a chain which conflicts with the finalised state but is otherwise entirely valid. They can feed you those blocks to lead you down the wrong chain.

However if your node was further along the chain to see one or more of those validators exit, you’d reject their attestations leaving less than 1/3 of the validators as dishonest and allowing you to reliably reach the real chain head.

So the weak subjectivity period is essentially how far behind your node can be before 1/3 of validators can have exited without you knowing about it. Once you  fall behind more than that, you need to confirm the chain you want to sync to out of band.

Introducing Pantheon

This week, the work I’ve been doing for the past 6 months, and that PegaSys has been working on for the past 18 months or so was released into the world. Specifically we’ve released Pantheon 0.8.1, our new MainNet compatible, Apache 2 licensed, Java-based Ethereum client. And it’s open source.

I’m pretty excited about it on a few fronts.  Firstly I think it’s a pretty important thing for the Ethereum community. To be a healthy ecosystem, Ethereum needs to have diversity in its clients to avoid a bug in one client taking out or accidentally hard forking the entire network. Currently though, Geth and Parity dominate the Ethereum client landscape.  Pantheon clearly won’t change that in the short term, but it is backed by significant engineering resources to help it keep up with the ever changing Ethereum landscape and be a dependable option.

I’m also really excited that Pantheon is released under the Apache 2.0 license. Both Parity and Geth along with most other clients are licensed under the GPL or LGPL. There are still a large number of enterprises that completely avoid the GPL and LGPL which has closed off Ethereum to them. Having Pantheon available under a permissive license and in a highly familiar language like Java will make it much easier for many enterprises to start using, building on and innovating with Ethereum.

Pantheon will also be building out functionality from the Enterprise Ethereum standard for things like privacy and permissioning to make private chains more powerful and flexible. Meanwhile we have a significant number of researches continuing to work on developing new ways to get the most out of Ethereum.

Finally I’m quite excited to be able to contribute to an open source project as my full time job. I’ve had the opportunity to do some open source for work in the past but only on a fairly small scale. It’s a bit daunting to have everything in the open and on the record, but I’m really looking forward to engaging with the community and being able to show exactly what I’ve been doing.