Ethereum Prysm client experiences mainnet incident, resource exhaustion leads to large-scale block and attestation loss

ChainCatcher news, the Prysm team released a post-mortem report on the mainnet incident, stating that during the Fusaka period on the Ethereum mainnet on December 4, almost all Prysm beacon nodes experienced resource exhaustion when processing specific attestations, resulting in an inability to respond to validator requests in a timely manner and causing a large number of missing blocks and attestations.

The impact range was from epoch 411439 to 411480, covering a total of 42 epochs. Out of 1344 slots, 248 blocks were missing, with a missing rate of about 18.5%. Network participation rate once dropped to 75%, and validators lost approximately 382 ETH in attestation rewards. The root cause was that Prysm received attestations from nodes that might have been out of sync with the mainnet, and these attestations referenced the block root of the previous epoch.

To verify their validity, Prysm repeatedly replayed the old epoch state and performed high-cost epoch transitions, causing nodes to trigger resource exhaustion under high concurrency. The related defect originated from Prysm PR 15965, which had been deployed to the testnet a month earlier but did not trigger the same scenario.

The official temporary solution is to enable the --disable-last-epoch-target parameter in version v7.0; the subsequently released v7.1 and v7.1.0 already include a long-term fix, which uses the head state to verify attestations and avoids repeatedly replaying historical states.

Prysm stated that the issue gradually eased after 4:45 UTC on December 4, and by epoch 411480, the network participation rate had recovered to over 95%.

The Prysm team pointed out that this incident highlights the importance of client diversity. If a single client accounts for more than one-third, it may lead to temporary inability to finalize; if it exceeds two-thirds, there is a risk of an invalid finalization chain. They also reflected on unclear communication regarding feature toggles and the inability of the test environment to simulate large-scale out-of-sync nodes, and will improve testing strategies and configuration management in the future.