Check why specific clusters with failed proposals have high consensus rounds

[boulder225]

🎯 Problem to be solved

Specific clusters, particularly those with failed proposals, exhibit unusually high consensus rounds, often exceeding three rounds. This pattern is especially pronounced in clusters assigned the builder_proposer duty. Additionally, there's an anomaly observed in the Daring Dolphin cluster, where despite having high latencies, it consistently achieves consensus in just one round. This contrasts with other clusters where a higher number of nodes on 'inc' correlates with problematic performance.

More in details:

• Analysis of failed proposals in clusters reveals high consensus rounds (3+), especially in the builder_proposer duty.
• Inconsistency observed in the Daring Dolphin cluster, with one round consensus but high latencies.
• Potential correlation between the number of nodes on 'inc' and the performance of clusters.
• Need to understand if this trend is due to randao, other factors, or mere coincidence.

The following query is focusing on consensus rounds by cluster name, peer, and duty. max(core_consensus_decided_rounds{cluster_name=~"$cluster_name",cluster_hash=~"$cluster_hash"}[$__interval]) by (cluster_name, cluster_peer,duty)

🛠️ Proposed solution

• [ ] analyse the builder_proposer duty across different clusters to identify any issues causing higher consensus rounds.
• [ ] Investigate the Daring Dolphin cluster's behaviour to understand why it deviates from the trend despite having high latencies.
• [ ] Examine the relationship between the number of nodes on 'inc' and cluster performance to determine if there is a causal link.
• [ ] Check if randao influences these trends