Audit and Fix our Flaky Tests

[fainashalts]

Many of our tests fail intermittently, and several people have had to increase timeouts to get them to pass CI. @benjamincburns says he has a list of tests, which he will add here to this issue.

[cds-amal]

Hey @benjamincburns do you have that list of tests?

[benjamincburns]

I had a look through my notes and couldn't find the list. I think however that this is likely data that we could mine from GitHub actions easily enough. Would be much easier to do so however if we made use of some structured data test reporter.

https://github.com/marketplace/actions/publish-unit-test-results https://www.npmjs.com/package/mocha-junit-reporter https://www.npmjs.com/package/jest-junit

[benjamincburns]

Just had some tests hang on this build: https://github.com/trufflesuite/truffle/actions/runs/2716306424 -- I think caused by a hang while retrieving a box for a test.

I strongly suspect that a lot of our test flakiness is due to the slowness of box retrieval. We should probably make an effort to remove as many box downloads as possible from our scenario test suite.

[benjamincburns]

I've also seen a handful of transient build failures recently due to HTTP errors while fetching packages from NPM.

[benjamincburns]

I had a look through my notes and couldn't find the list. I think however that this is likely data that we could mine from GitHub actions easily enough. Would be much easier to do so however if we made use of some structured data test reporter.

https://github.com/marketplace/actions/publish-unit-test-results https://www.npmjs.com/package/mocha-junit-reporter https://www.npmjs.com/package/jest-junit

I created #5345 to account for this

[benjamincburns]

Flaky test:

truffle:   150 passing (18m)
truffle:   1 failing
truffle:   1) Scenarios
truffle:        Cyclic Dependencies [ @standalone ]
truffle:          compiles cyclic dependencies that Solidity is fine with (no `new`'s):
truffle:      Error: Timeout of 20000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/runner/work/truffle/truffle/packages/truffle/test/scenarios.js)
truffle:       at listOnTimeout (node:internal/timers:559:17)
truffle:       at processTimers (node:internal/timers:502:7)
truffle: 
truffle: error Command failed with exit code 1.

https://github.com/trufflesuite/truffle/runs/7538045531

[benjamincburns]

Flaky test:

@truffle/debugger:   1) Source mapping (location and jumps)
@truffle/debugger:        Function Depth
@truffle/debugger:          counts each external call for 1, not 0 or 2:
@truffle/debugger:      Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/runner/work/truffle/truffle/packages/debugger/dist/debugger.js)
@truffle/debugger:       at runMicrotasks (<anonymous>)
@truffle/debugger:       at processTicksAndRejections (internal/process/task_queues.js:95:5)

https://github.com/trufflesuite/truffle/runs/7685133626?check_suite_focus=true

[benjamincburns]

Flaky test:

@truffle/core:   1 failing
@truffle/core:   1) NPM integration
@truffle/core:        "before all" hook: Create a sandbox for "successfully finds the correct source via Sources lookup":
@truffle/core:      Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/runner/work/truffle/truffle/packages/core/test/npm.js)
@truffle/core:       at listOnTimeout (node:internal/timers:559:17)
@truffle/core:       at processTimers (node:internal/timers:502:7)
@truffle/core: error Command failed with exit code 1.

https://github.com/trufflesuite/truffle/runs/7685126493?check_suite_focus=true

[benjamincburns]

Flaky test:

@truffle/db: FAIL src/network/test/index.spec.ts (26.55 s)
@truffle/db:   Network
@truffle/db:     for arbitrary batches of blocks from an arbitrary set of arbitrarily forked blockchains
@truffle/db:       ✕ saves network genealogies so that @truffle/db correctly reports known latest descendant networks (numRuns: 50) (with seed=1660894803295) (25089 ms)
@truffle/db:   ● Network › for arbitrary batches of blocks from an arbitrary set of arbitrarily forked blockchains › saves network genealogies so that @truffle/db correctly reports known latest descendant networks (numRuns: 50) (with seed=1660894803295)

https://github.com/trufflesuite/truffle/runs/7913565921?check_suite_focus=true

[benjamincburns]

Hung tests (had to cancel job):

https://github.com/trufflesuite/truffle/runs/7914744801?check_suite_focus=true

[benjamincburns]

Flaky test:

@truffle/compile-solidity:   1) CompilerSupplier
@truffle/compile-solidity:        integration
@truffle/compile-solidity:          compiles w/ remote solc when options specify release (pinned):
@truffle/compile-solidity:      Error: Timeout of 40000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

https://github.com/trufflesuite/truffle/runs/8009652049?check_suite_focus=true