At slot 281, Prysm cannot import a Teku built block. The message we get is

[2026-03-04 15:28:11.08] DEBUG sync: Could not validate beacon block error=could not get pre state for slot 281: could not process block: could not process execution payload bid: bid consistency validation failed: bid parent block hash mismatch: got d1424c208f74aec1cd64c25f18745bc964b2f4bdbe075fe9c9d3d890d125973d, expected 03611a4c6839eb3dc9ca69c49a26b387e3a3a3a6ce951086620b55d723a5c6aa graffiti=teku-geth-1 GE57c3TK5704 proposerIndex=1826 slot=281 version=7

The bids corresponds to these blocks:

In slot 223 we get

[2026-03-04 15:16:35.16]  INFO blockchain: Synced new block block=0xb1218931... blockHash=0x03611a4c6839 builderIndex=18446744073709551615 chainServiceProcessedTime=32.168967ms epoch=6 finalizedEpoch=4 finalizedRoot=0xe0d13059... justifiedEpoch=5 justifiedRoot=0x6444154d... parentHash=0x0647de063efc parentRoot=0x12574d3d... sinceSlotStartTime=167.984196ms slot=223 slotInEpoch=31 version=gloas

And In slot 224

[2026-03-04 15:16:47.13]  INFO blockchain: Synced new block block=0x22d0ecee... blockHash=0xd1424c208f74 builderIndex=18446744073709551615 chainServiceProcessedTime=22.696689ms epoch=7 finalizedEpoch=5 finalizedRoot=0x6444154d... justifiedEpoch=6 justifiedRoot=0xdf424ae5... parentHash=0x03611a4c6839 parentRoot=0xb1218931... sinceSlotStartTime=135.882585ms slot=224 slotInEpoch=0 version=gloas

[2026-03-04 15:16:47.16]  INFO blockchain: Processed execution payload envelope blockHash=0xd1424c208f74 blockRoot=0x22d0eceea54f parentHash=0x03611a4c6839 slot=224

So it looks as though 224 built on top of full 223, and this bad lead led me on a wrong path. The error message on the bid consistency validation makes it appear as though Teku’s block was being processed with the empty state of 224 since we expected to see 223’s hash as parent hash and we obtained instead the blockhash of 224, we fail processing Teku’s block and we declare it as invalid.

And this was my working hypothesis for a long time, except no one could give me Teku’s block to check the hypothesis. There was something weird though: I could not explain the error message chain. That error message for the bid validation would come at block processing, not block gossiping!

The status of FFG at the time

The chain was finalizing fine every epoch at the time. Blocks received during that time reported something like:

[2026-03-04 15:27:59.06]  INFO blockchain: Synced new block block=0x64f09aa9... blockHash=0xc707efd315d3 builderInde
x=18446744073709551615 chainServiceProcessedTime=24.738337ms epoch=8 finalizedEpoch=6 finalizedRoot=0xdf424ae5... ju
stifiedEpoch=7 justifiedRoot=0x22d0ecee... parentHash=0xd85ae68ef2f9 parentRoot=0xd6555ba8... sinceSlotStartTime=63.
887843ms slot=280 slotInEpoch=24 version=gloas

So Slot 280 in epoch 8 had already epoch 6 as finalized epoch. That means the finalized checkpoint root corresponds to slot 192, and since this slot onwards all blocks are in forkchoice. The justified checkpoint is at slot 224, precisely where I thought Teku was building on (which also I thought they were probably stuck in Forkchoice that tends to then take the justified checkpoint as head in this case). We should import Teku’s block since it may actually become head; for some miraculous reason that I never really understood we are forced to sync these blocks over gossip.

The way to pick the parent state is to look up in forkchoice the parent block root, that block had a committed blockhash (in this case 0xd1424...) which we compare to the incoming’s bid’s parent hash. If they are equal, this block builds on top of full and that’s the state we need to pick up. Typically this state would be cached, it’s either head, or the previous one and so forth. But this block was being based 57 slots ago so it was not in our typical caches for head or hot states. Here is where the error message from Teku’s block is quite interesting and misleading and it took me a long time to find the actual bug.

Our prestate fetcher has received Teku’s block for 281, it sends to lookup the full poststate of block+payload of slot 224. But if the block was actually based on 224 then the poststate of it would have been cached, not once bbut in many different caches. It’s the damned justified checkpoint. Then it hit me, right before the error message there’s this one that I had dismissed

[2026-03-04 15:28:11.08] DEBUG stategen: Replaying state diff=27 endSlot=251 startSlot=224

And here’s where I recognized what’s happened. In fact Teku’s block must have been based on 251, not 224 and that’s why to get the pre-state we need to process up to that slot. To get that state we grab our last boundary state, which is 224 in this case, and start processing blocks from it. Now here is where prysm fails, bizarrely, this commit was supposedly pushed the previous night to the devnet branch, but for some reason it wasn’t. The stategen replayer did not account for payloads, so it failed processing the very first block, as it started applying the beacon block of 225 instead of the payload! So the error is from incorrectly trying to apply 225 on top of 224, the hashes in the bid message had nothing to do with Teku’s block.

We got confirmation from @0xunclebill at lighthouse that indeed Teku’s block was based on 251 and was valid.

Downscoring peers

Once Prysm declared Teku’s block as invalid, the chain continued finalizing for a few epochs. This is because Lighthouse, Lodestar and Prysm kept producing blocks on the canonical chain. However, Lighthouse and Lodestar’s behavior towards Teku was very different than Prysm. Prym immediately disconnected from Teku as it was proposing blocks in what it considered an invalid branch, but Lodestar and Lighthouse kept propagating blocks and payloads from that branch. What is still a mistery to me is why we kept getting the payload envelope for slot 281 over gossip!

At any rate, since Lighthouse and Lodestar kept propagating things that we could not validate as we didn’t have the parent, we kept ignoring them happily.

However, this is where the situation compounds with another known bug that had already been fixed in Prysm, but the image deployed in the devnet did not have this fix. When Prysm gets a payload that it cannot validate because it points to an unknown beacon block root. It validates the builder’s signature before putting it on a pending queue. For external builders (those that are not being currently tested) this is relatively safe to do with the head state (there are some edge cases though that after this event we may need to reevaluate). But for self-built payloads, it requires to have a reasonable state that has the right proposer index for the given slot, in order to validate the signature. What we should do for self built payloads is to just ignore them if they fail the signature, and keep them if they pass. But what we did in the devnet branch is reject them if they fail the signature check.

What happened is that Teku was already iin it’s own branch, so there were 2 proposers at the same slot. Teku kept proposing payloads, which our peers sent over gossip. And we rejected them right away cause we didn’t have the beacon block. We couldn’t possibly have the beacon block cause it’s based on invalid according to our previous bug. So we downscored our peers. After a while we were completely isolated.

To make things worse, our peers would also send us Teku’s blocks, which we had to put on a pending queue cause we didn’t have the parent payload, which we request by root and add for later processing. But that also led to downscoring them for giving us bad data!

What we learned

We should have included both fixes in the devnet branch, but actually they still need to be tweaked!.