The node keeps a list of whitelisted and blacklisted builders, they serve different purposes. Blacklisted builders will have their bids rejected and may not be contacted by direct connections if they are known. Whitelisted builders on the other hand are builders that will receive preferential treatment even in the event of a global circuit breaker triggering.

In gloas there are three types of builders, those over P2P that can be largely anonymous, and those being reached by direct connections that can be trusted or not. It is unclear if there will be untrusted direct connection builders on the market, but we should prepare our circuit breaker as if they existed. Untrusted builders can deploy multiple keys, rotate them, try to attack the payload market, etc. Trusted builders can do so as well, but they are well established institutions with market to lose. So in some circuit breaker scenarios, it is actually convenient to fall back to trusted builders rather than falling back to self-building immediately. There are very little scenarios in which falling back to p2p bids is preferable.

Expected circuit breaker scenarios

There are three situations in which one would expect the most likely reason to trigger a circuit breaker.

A large builder bug

This has been the main reason for missing payloads/blocks on mainnet. A large builder wins the auction but it fails to deliver the payload. We can be very aggressive to blacklist builders in this situation. The rationale is as follows. First of all we can immediately blacklist the corresponding builder index. If this builder index is for a P2P bidder, then it is not expected to be a large builder anyway. In this type of scenarios we want to blacklist a single builder or a single relay.

Client bugs

This has happened on mainnet in which CL clients failed to produce enough blocks triggering a cicuit breaker. This becomes most likely not an issue on Gloas, in fact clients should trigger the circuit breaker on payloads missing rather than consensus blocks (and therefore entire slots) missing. In this type of scenarios we don’t need to do anything.

A builder attack on the network

The most likely scenario expected for this would come from untrusted builders, that either flood the P2P stack with winning bids that don’t deliver payloads or similar liveness situations. In this scenario we want to first fallback to the network of trusted builders and if this fallback fails, only then we want to fallback to self-building.

Operator configuration.

The operator can specify at the beacon node level a list of whitelisted and a list of blacklisted builder indices. Documentation of the feature needs to be exhaustive since builder indices are reused and this list may go stale appropriately. Operators need to be conservative and make sure that the list is either kept up to date, or only add builders that are seriously trusted.

Alternatively, if large builders were to help, the beacon node could keep a list of possible builder indices associated to each direct connection URL, so as to be able to whitelist/blacklist per URL rather than builder index.

Beacon node tracking

Failed builders

The beacon node will keep a list of builders with failures

type FailedBuilder struct {
    index           BuilderIndex
    failed          uint8
    whitelistEpoch  Epoch 
    backOffEpoch    Epoch

}

The failed entry keeps track of how many consecutive payloads were not delivered by this builder. The whitelistEpoch is set to a non-zero value when this builder index is blacklisted, in which case this field indicates when this builder should be whitelisted again. The field backOffEpoch has a similar flavor, it indicates at which epoch the failed counter should be zeroed out again. This allows to keep tracking failures of whitelisted builders so as if they continue to fail they can be blacklisted again immediately and for longer times.

Relays

The beacon node will keep track of bids returned by contacted endpoints on direct connections. These are objects like

type RelayBuilder struct {
    identifier      string
    indices         []BuilderIndex
}

The identifier string could be the URL contacted or a proxy data field. It has to be the same data that is passed to the beacon by the validator client to make the direct connection. The indices are the builder indices seen from bids returned on that endpoint. The operator can modify these lists by passing on the command line expected indices for a given relay identifier. These lists can be mantained by the community. This list is mostly useless for a homestaker that will rarely propose but it is relevant for a staking pool operator.

The point of this list is that every time that a payload is failed to be delivered by a builder index, an entry in the FailedBuilder list is added/updated, but also another entry is created by any other builder that is served by the same relay, so as to ban the relay as if it were a single builder.

Extra constants

Tolerance

The operator could specify a constant AllowedFailures to specify how many failures a given builder is allowed to have before being blacklisted. This could be specified per-builder, relay, builder-group etc. But I will write this as if this were a global constant (I believe this should default to 0).

Another constant is CriticalFailures which specifies that this builder should be blacklisted even if it was whitelisted before. The usage of this constant will be explained below.

Back off period

The constant BackOffPeriod measures how many epochs without a bid coming from the given builder will trigger the failed counter to be reset to 0. The constant WhitelistPeriod measures how many epochs the builder will remain blakclisted on the first offense above AllowedFailures.

Systemic failure

We have a treshold of CriticalFailedBuilders and CriticalFailedUntrustedBuilders after which we fallback to either trusted builders or self-building

Operation

Single builder banning

Whenever a payload is missing and the corresponding beacon block has been attested by over 60% of the chain, an entry for the corresponding builder in the FailedBuilder is created or updated.

If an entry for that builder already exists, then we increment the failed counter. If the builder attains a number of failures above AllowedFailures then we set its whiteListEpoch to the current epoch plus WhitelistPeriod and we set its backOffEpoch to the current epoch plus BackOffPeriod. If the number of failures attains CriticalFailures then we can set the whitelist epoch or the backoff epoch even further higher, to a much higher constant or simply until the operator restarts the node (unless the builder broadcasts a valid payload and the failed counter resets to zero.

A typical configuration could be with AllowedFailures = 0, CriticalFailures=2, WhitelistPeriod=1 and BackOffPeriod=5 so that on a single failure we blacklist the builder for just one epoch, but on two consecutive failures during 5 epochs the builder is permanently banned until a restart.

If the failed builder is part of the indices in one of the tracked RelayBuilder objects, the same procedure is applied to all builders in that list. For example, if the node tracks a relay as a RelayBuilder with indices = [1,2] and the builder 1 fails to broadcast a payload. The node will track a FailedBuilder with index 1 as above, with failed=1 and another with index 2 and failed=0, but both possibly blacklisted, even if failed=0.

Single builder whitelisting

if a payload is seen from a builder in the failed list, we immediately whitelist it and remove it from the list (unless it’s part of a relay). This could be further designed to keep a quarantine to see if it fails again and track it in a special manner, but this seemed overkill to me.

If the builder is part of a relay, we simply put failed=0, when all members of the same relay have failed=0 we whitelist them all.

Systemic failures

Whenever the node is trackingCriticalFaildeUntrustedBuilders or more with a whitelistEpoch higher than the current one (so that they are currently blacklisted) then the node does no longer take P2P bids and falls back to only bids that are requested from trusted builders (that can be the direct connections or the whitelisted indices that were specified by the operator at launch).

Whenever the node achieves CriticalFailedBuilders or more that are blacklisted, then the node falls back to self-building.

Notice that this is no longer tracking slots missed in an epoch or consecutive slots, both these metrics should be already covered by simply tracking the number of banned builders and being conservative enough with this metric.

Reasonable defaults for these are CriticalFailedBuilders = 7 and CriticalFaildeUntrustedBuilders = 3.

Conclusion

With the above design and the most common scenarios mentioned above we would have the following results

Under a single large builder bug, that builder would be immediately banned for an epoch, limiting the damage to just a single block during the epoch, if the builder is still buggy during the next epoch it will be permanently blacklisted. No damange to the network otherwise.

If there is a client bug preventing blocks, no changes to the circuit breaker are triggered.

If there is an attack from untrusted builders, then they would lose 3 bids and then nodes fallback to trusted builders.

Of course this works as is if everyone uses a similar design. If nodes use completely separate designs, still the defaults are reasolable enough so that Prysm nodes will fallback first to trusted builders and only in critical situations to self-building. At the same time, the backoff mechanism allows them to reasonably retry a builder if it hasn’t won a bid in a while after a failure.