[Consensus] Timeout objects rebroadcast

[durkmurder]

https://github.com/dapperlabs/flow-go/issues/6385

Context

This PR implements rebroadcast of timeout objects. I have chosen an implementation which encapsulates rebroadcast logic in timeout.Controller, meaning ActivePaceMaker, EventLoop don't know anything about concept of rebroadcast, they just react to events in timeout channel and create timeout, broadcast it when asked to. This approach seems clean and ergonomic, the biggest challenge was to implement single-shot timeout + tick events over the timeout channel.

Notable changes:

• Controller spawns a goroutine and channel for every round to send timeout events.
• Updated integration tests with new run/stop logic.
• Made the maximum rebroadcast interval configurable.
• Updated compliance engine to wait for spawned goroutines.

[codecov-commenter]

Codecov Report

Merging #3277 (736d230) into feature/active-pacemaker (42c7f37) will decrease coverage by 4.79%. The diff coverage is 43.22%.

@@                     Coverage Diff                      @@
##           feature/active-pacemaker    #3277      +/-   ##
============================================================
- Coverage                     55.29%   50.50%   -4.80%     
============================================================
  Files                           748      328     -420     
  Lines                         69351    31629   -37722     
============================================================
- Hits                          38349    15973   -22376     
+ Misses                        27875    14282   -13593     
+ Partials                       3127     1374    -1753     

Flag	Coverage Δ
unittests	50.50% <43.22%> (-4.80%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
consensus/config.go	0.00% <ø> (ø)
consensus/participant.go	18.70% <0.00%> (-0.28%)	:arrow_down:
consensus/recovery/participant.go	0.00% <0.00%> (ø)
consensus/hotstuff/pacemaker/timeout/controller.go	50.00% <5.71%> (-19.10%)	:arrow_down:
...nsensus/hotstuff/voteaggregator/vote_aggregator.go	55.17% <49.38%> (+14.60%)	:arrow_up:
consensus/hotstuff/pacemaker/pacemaker.go	74.13% <72.72%> (-1.76%)	:arrow_down:
consensus/hotstuff/pacemaker/timeout/config.go	88.00% <76.92%> (-5.19%)	:arrow_down:
consensus/hotstuff/eventhandler/event_handler.go	75.07% <100.00%> (+0.97%)	:arrow_up:
consensus/hotstuff/eventloop/event_loop.go	75.59% <100.00%> (-0.12%)	:arrow_down:
module/util/util.go	86.51% <100.00%> (-0.86%)	:arrow_down:
... and 426 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[AlexHentschel]

A cleanliness point (unrelated to this PR): would you mind adding a brief comment about error returns to the implementation here: https://github.com/onflow/flow-go/blob/207e42764f811f65861596e76e9e11ea1dcaa6af/consensus/hotstuff/votecollector/statemachine.go#L176 the interface states:

	// It returns nil if the block is valid.
	// It returns model.InvalidBlockError if block is invalid.
	// It returns other error if there is exception processing the block.

which I feel would be valuable to also add to the implementation

[durkmurder]

bors merge

[bors[bot]]

Build succeeded:

• Integration Tests (make -C integration access-tests)
• Integration Tests (make -C integration bft-tests)
• Integration Tests (make -C integration collection-tests)
• Integration Tests (make -C integration consensus-tests)
• Integration Tests (make -C integration epochs-tests)
• Integration Tests (make -C integration execution-tests)
• Integration Tests (make -C integration ghost-tests)
• Integration Tests (make -C integration mvp-tests)
• Integration Tests (make -C integration network-tests)
• Integration Tests (make -C integration verification-tests)
• Lint (./)
• Lint (./crypto/)
• Lint (./integration/)
• Unit Tests (access)
• Unit Tests (admin)
• Unit Tests (cmd)
• Unit Tests (consensus)
• Unit Tests (engine)
• Unit Tests (fvm)
• Unit Tests (ledger)
• Unit Tests (module)
• Unit Tests (network)
• Unit Tests (others)
• Unit Tests (utils)