Block size is growing

[HarukaMa]

👉 Please follow one of these issue templates 👈

Note: to keep the backlog clean and actionable, issues may be immediately closed if they do not follow one of the above issue templates.

We don't have discussion enabled in this repo and this doesn't apply to any of the templates (IMO), but I still want to have a place to track this as the devs telegram group is not really for async discussions - messages are forgotten pretty quickly.

As we already have 60 validators on the network (as of height ~970000), the subdag within each block is growing a lot bigger - in fact, it's currently the largest component of a block. This has already caused slower syncing for nodes (5-6 blk/s around height 700000, and 2-3 blk/s around height 920000) and fast growing storage space usage. Although we have the 200 upper limit, this growth is not linear AFAICT.

Also, when syncing now, there are a lot of Candidate block ... from '...' is malformed messages, and this indicates different nodes are having slightly different blocks. After closer inspection, I found the difference is in election_certificate_ids field in subdag:

For the same block:

there are different views:

The remaining parts are the same so the network still runs, but at least for now, this is making the p2p connection inefficient as the "malformed" message will cause a disconnection. Also, I'm not sure if this is intended - shouldn't all nodes have the exact same block view?

[HarukaMa]

Update: seems block uniqueness is not an issue here, and the block size issue can be partly resolved by #3000.