flaky test: test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline

[PSeitz]

failures:
    actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 159 filtered out; finished in 1.03s


--- TRY 2 STDERR:        quickwit-indexing actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline ---
thread 'actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline' panicked at quickwit-indexing/src/actors/indexing_service.rs:1532:9:
assertion failed: universe.get_one::<MergePipeline>().is_none()
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: quickwit_indexing::actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline::{{closure}}
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
   6: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
   7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}0
   8: tokio::runtime::scheduler::current_thread::Context::ente1
   9: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}2
  10: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}3
  11: tokio::runtime::context::scoped::Scoped<T>::se4
  12: tokio::runtime::context::set_scheduler::{{closure}5
  13: std::thread::local::LocalKey<T>::try_wit6
  14: std::thread::local::LocalKey<T>::wit7
  15: tokio::runtime::context::set_schedule8
  16: tokio::runtime::scheduler::current_thread::CoreGuard::ente9
  17: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
  18: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
  19: tokio::runtime::context::runtime::enter_runtime
  20: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
  21: tokio::runtime::runtime::Runtime::block_on_inner
  22: tokio::runtime::runtime::Runtime::block_on
  23: quickwit_indexing::actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline
  24: quickwit_indexing::actors::indexing_service::tests::test_indexing_service_shutdown_merge_pipeline_when_no_indexing_pipeline::{{closure}}
  25: core::ops::function::FnOnce::call_once

May be related: I also noted that shutting down quickwit (current main) leads to a hanging:

2024-10-18T04:37:34.506Z  INFO quickwit_janitor::actors::garbage_collector: loaded 4 indexes from the metastore
2024-10-18T04:43:30.479Z  INFO quickwit_ingest::ingest_v2::idle: closed idle shard `test-index-pattern-777:01JAEY43PWZ83AKRXXQHRFVBXN/_ingest-source/01JAEY43R75AX2JN7S14EAYVRD`
❢ Graceful shutdown initiated. Waiting for ingested data to be indexed. This may take a few minutes. Press Ctrl+C again to force shutdown.
2024-10-18T04:46:25.609Z  INFO quickwit_ingest::ingest_v2::ingester: decommissioning ingester
2024-10-18T04:47:34.525Z  INFO quickwit_janitor::actors::garbage_collector: loaded 4 indexes from the metastore