Investigate high number of orphaned blocks

[deepthiskumar]

68% of blocks were non-canonical in the during July-Sept 2024 which seems a bit high It is possible for two or more BPs to win a slot, this is encoded in the consensus and is to provide some redundancies. One other reason (potentially a bigger contributor) is duplicate BPs- A lot of BPs run multiple nodes with the same key to ensure at least one of them makes it. Could node operators stop running duplicates? or reduce the count?

More data required-

1. How much of the 68% is due to duplicate blocks
2. For BP keys that run duplicate BPs, do all their blocks still get produced and gossiped? (if so, then node reliability has improved and duplicate nodes may not be required anymore. The need to run redundant infra adds to node operator cost
3. Are there disconnected blocks amongst the orphans? Disconnected blocks- If a block reaches a BP past slot time and a node produces the next block off the parent of the latest block, the produced block will be disconnected from the what the netowkr considered canonical chain at the time. The network will then consider such disconnected block as forks, eventually they'll be orphaned if they do not grow stronger. Caused by nodes being busy causing delays in block propagation, possibly due to graphql requests blocking the node during payments or processing other network messages is blocking block validation