Outdated Node Clients Cause Blockchain Reorg on Eth2 Beacon Chain

The Eth2 Beacon Chain’s validators fell out of step on May 25 after a client update boosted some clients but caused confusion among validators who hadn’t bothered to upgrade.

The incident, called a blockchain reorganization, happens when nodes disagree on the order of the most recent block. Blockchains order blocks – batches of transactions – chronologically. 

If some nodes are faster than others, they can’t agree which block should come first. If that happens, they’ll keep adding blocks to their own version of the blockchain, then discard the shorter chain when the next block gets produced.

That’s exactly what happened yesterday. 

Beacon Chain clients had staggered the release of upgrades that sped up block submissions, but about 25% of validators hadn’t updated their software despite having weeks to do so, said Ethereum core developer Terence Tsao on Twitter.

At block 3,887,074, the up-to-date nodes were about 12 seconds faster than validators who hadn’t updated their clients. The upgraded clients submitted the next block earlier than the rest of the validators, sparking confusion about whose block should come first.

The Beacon Chain didn’t know what to do until an old client proposed a new block, prompting a reorg. This happened six more times until the validators finally agreed on the correct order of the chain.

Nothing too bad happened this time around – the Beacon Chain only supports ETH2 staking; it doesn’t support peer-to-peer transactions yet. 

But if the chain reorganized after Ethereum’s forthcoming chain-merge, nodes might have rejected transactions while they tried to agree on the order of the most recent blocks.

Ethereum co-founder Gavin Wood wrote in 2015 that during reorgs, the most recent transactions are reverted, and “the transactions in the newer replacement are executed.”

He said that, since Ethereum targets a block time of about 15 seconds, “this actually happens naturally rather often.” 

Tsao said yesterday’s reorg is unlikely to happen again, so long as the outdated nodes update their clients.