Why is `layer-1-tests (tests::l1_observer_test::l1_integration_test)` flaky?

[obycode]

Describe the bug This test tends to fail in CI but pass locally, and will pass in CI sometimes. We need to identify why that is and then fix it.

Steps To Reproduce Trigger a CI by opening a PR. You may see the test fail. For an example of a failing instance, see https://github.com/hirosystems/stacks-subnets/actions/runs/5479213207/jobs/9981342347?pr=314

Expected behavior These tests should all pass reliably.