Various tests sometimes fail

[upbqdn]

Description

I noticed four issues when running the tests for the main branch locally on my machine:

• the test zebra_rpc_conflict often fails with:

  conflicted node2 was still running, but the test expected a panic
  

• the test restart_stop_at_height sometimes hangs even for Mainnet. By hanging I mean that the test doesn't finish in two hours, and htop shows no activity regarding Zebra.
• the test activate_mempool_mainnet sometimes fails with

  0: stdout of command did not contain any matches for the given regex
  ...
  Match Regex:
   [
       "enabling mempool for debugging",
   ]
  

• the test peer_set::candidate_set::tests::prop::skipping_outbound_peer_connection_skips_rate_limit often fails with:

  Message:  assertion failed: `(left == right)`
    left: `Err(Elapsed(()))`,
   right: `Ok(None)`
  Location: zebra-network/src/peer_set/candidate_set/tests/prop.rs:79
  

[teor2345]

All of these could be timing issues, here's what I'd suggest to start investigating:

1. Try increasing the delay between node launches
2. Try increasing the delay before the restart
   • We should get better diagnostics for this once we fix #4591
   • You can replace zebrad... with timelimit -T (time) zebrad... to get the process to exit and show its logs
3. Try increasing the zebrad command timeout
4. Elapsed means a tokio timeout, try increasing the timeout that's failing

[teor2345]

Some of these bugs might be fixed by:

• #4747

[teor2345]

We've made a bunch of fixes to these tests recently, feel free to open a separate ticket for each test that is still failing after we finish the fixes in the next Zebra release.