Test flake: timeout exceeded waiting for block

[rootulp]

Summary of Bug

I am observing that the latest commit on master has a test that failed CI: https://github.com/celestiaorg/celestia-app/runs/7550586435?check_suite_focus=true#step:6:133

--- FAIL: TestIntegrationTestSuite (77.44s)
    block_size_test.go:44: setting up integration test suite
    network.go:203: acquiring test network lock
    network.go:213: preparing test network with chain-id "chain-4etufu"
    network.go:484: starting test network...
    network.go:490: started validator 0
    network.go:490: started validator 1
    network.go:498: started test network at height: 0
    --- FAIL: TestIntegrationTestSuite/TestMaxBlockSize (48.83s)
        block_size_test.go:125: 
            	Error Trace:	/home/runner/work/celestia-app/celestia-app/app/test/block_size_test.go:125
            	            				/home/runner/work/celestia-app/celestia-app/app/test/suite.go:91
            	Error:      	Received unexpected error:
            	            	timeout exceeded waiting for block
            	Test:       	TestIntegrationTestSuite/TestMaxBlockSize
        --- FAIL: TestIntegrationTestSuite/TestMaxBlockSize/100_random_txs (11.20s)
            testing.go:[133](https://github.com/celestiaorg/celestia-app/runs/7550586435?check_suite_focus=true#step:6:134)6: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
    --- FAIL: TestIntegrationTestSuite/TestSubmitWirePayForData (10.03s)
        block_size_test.go:215: 
            	Error Trace:	/home/runner/work/celestia-app/celestia-app/app/test/block_size_test.go:215
            	            				/home/runner/work/celestia-app/celestia-app/app/test/suite.go:91
            	Error:      	Not equal: 
            	            	expected: 0x0
            	            	actual  : 0x20
            	Test:       	TestIntegrationTestSuite/TestSubmitWirePayForData
        block_size_test.go:216: 
            	Error Trace:	/home/runner/work/celestia-app/celestia-app/app/test/block_size_test.go:216
            	            				/home/runner/work/celestia-app/celestia-app/app/test/suite.go:91
            	Error:      	Received unexpected error:
            	            	timeout exceeded waiting for block
            	Test:       	TestIntegrationTestSuite/TestSubmitWirePayForData
        --- FAIL: TestIntegrationTestSuite/TestSubmitWirePayForData/small_random_typical (10.01s)
            testing.go:1336: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
    block_size_test.go:70: tearing down integration test suite
    network.go:587: cleaning up test network...
    network.go:614: finished cleaning up test network
    network.go:584: released test network lock

Version

https://github.com/celestiaorg/celestia-app/commit/c3a200b9ea8aafe1f56c555f2faa175b7f051612

Steps to Reproduce

CI produced this error however I assume this test is flakey because this check passed in https://github.com/celestiaorg/celestia-app/pull/570 prior to merge

For Admin Use

• [ ] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged
• [ ] Contributor assigned/self-assigned

[rootulp]

Another instance of this flake: https://github.com/celestiaorg/celestia-app/actions/runs/2949245994/attempts/1

[rootulp]

Possible cause which uses a sleep timeout from cosmos-sdk https://github.com/celestiaorg/celestia-app/blob/40d195b9ba89280bab61b119f9863b6b074d6d34/app/test/block_size_test.go#L125

Ref:

• https://github.com/teivah/100-go-mistakes#sleeping-in-unit-tests-86

[evan-forbes]

Possible cause which uses a sleep timeout from cosmos-sdk

this makes sense, and I think could be helped by decreasing the timeout commit after using the new testnode #824 and switching to network.WaitForHeight instead of repeatedly calling network.WaitForNextBlock

[rootulp]

This comment is misleading: https://github.com/rootulp/celestia-app/blob/841cc147e5ba5b4621763e9ae1cc748fd5f044d5/app/test/integration_test.go#L116-L119

because that generator uses accounts 40 - 120 which doesn't overlap with the previous generator.