Evidence may not work consistently

[faddat]

Evidence may not work consistently

Note: this issue is adjacent to #1948, but different from it in that this issue raises the suspicion that there are issues with evidence itself, as opposed to the test.

Recently I've evolved testbyzantineprevoteequivocation significantly which:

• gave better insight into the test
• improved frequency of passage significantly
• did not eliminate failures completely

The next steps are to continue to explore issues with #1917 and if unsuccessful in resolving the problem in #1917, coming here to work on the code for evidence itself instead of the tests.

I've been working on making the tests consistent, and have largely been successful:

--- FAIL: TestByzantinePrevoteEquivocation (10.04s)
    byzantine_prevote_equivocation_test.go:44: Generating a random genesis document and private validators
    byzantine_prevote_equivocation_test.go:49: Initializing test environment
    byzantine_prevote_equivocation_test.go:50: Initializing validators
    byzantine_prevote_equivocation_test.go:50: Initializing reactors for each validator
    byzantine_prevote_equivocation_test.go:53: Making connected switches and starting all reactors
    byzantine_prevote_equivocation_test.go:60: Creating byzantine validator
    byzantine_prevote_equivocation_test.go:64: Altering prevote so that the byzantine node double votes when height is 2
    byzantine_prevote_equivocation_test.go:68: Starting the consensus reactors
    byzantine_prevote_equivocation_test.go:72: Collecting evidence from each validator
    byzantine_prevote_equivocation_test.go:455: Timeout waiting for evidence from validator 0
    byzantine_prevote_equivocation_test.go:455: Timeout waiting for evidence from validator 2
    byzantine_prevote_equivocation_test.go:455: Timeout waiting for evidence from validator 3
    byzantine_prevote_equivocation_test.go:455: Timeout waiting for evidence from validator 1
    byzantine_prevote_equivocation_test.go:73: No evidence was collected from any validator
    byzantine_prevote_equivocation_test.go:74: 
        	Error Trace:	/home/runner/work/cometbft/cometbft/internal/consensus/byzantine_prevote_equivocation_test.go:74
        	Error:      	Received unexpected error:
        	            	no evidence collected from validators
        	Test:       	TestByzantinePrevoteEquivocation
        	Messages:   	Failed to collect evidence from each validator
FAIL
coverage: 78.3% of statements

Discovered by: https://github.com/cometbft/cometbft/pull/2005

Now, if tbpe fails, we get pretty specific information as to why that is.

I think that there is an issue with evidence submission in comet.