Flaky cli tests

[PSeitz]

Executed cargo test

test test_delete_index_cli ... FAILED
test test_cmd_search_aggregation ... FAILED
test test_all_local_index ... FAILED
test test_ingest_docs_cli ... FAILED
test test_garbage_collect_index_cli ... FAILED
test test_garbage_collect_cli_no_grace ... FAILED

failures:

---- test_delete_index_cli stdout ----
thread 'test_delete_index_cli' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Failed to connect to `127.0.0.1:40785` failed, retrying 1/10
Failed to connect to `127.0.0.1:40785` failed, retrying 2/10
Failed to connect to `127.0.0.1:40785` failed, retrying 3/10
Failed to connect to `127.0.0.1:40785` failed, retrying 4/10
Failed to connect to `127.0.0.1:40785` failed, retrying 5/10
Failed to connect to `127.0.0.1:40785` failed, retrying 6/10
Failed to connect to `127.0.0.1:40785` failed, retrying 7/10
Failed to connect to `127.0.0.1:40785` failed, retrying 8/10
Failed to connect to `127.0.0.1:40785` failed, retrying 9/10
Failed to connect to `127.0.0.1:40785` failed, retrying 10/10
thread 'test_delete_index_cli' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:40785`', quickwit-cli/tests/cli.rs:608:35

---- test_cmd_search_aggregation stdout ----
thread 'test_cmd_search_aggregation' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:43953` failed, retrying 1/10
Failed to connect to `127.0.0.1:43953` failed, retrying 2/10
Failed to connect to `127.0.0.1:43953` failed, retrying 3/10
Failed to connect to `127.0.0.1:43953` failed, retrying 4/10
Failed to connect to `127.0.0.1:43953` failed, retrying 5/10
Failed to connect to `127.0.0.1:43953` failed, retrying 6/10
Failed to connect to `127.0.0.1:43953` failed, retrying 7/10
Failed to connect to `127.0.0.1:43953` failed, retrying 8/10
Failed to connect to `127.0.0.1:43953` failed, retrying 9/10
Failed to connect to `127.0.0.1:43953` failed, retrying 10/10
thread 'test_cmd_search_aggregation' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:43953`', quickwit-cli/tests/cli.rs:343:35

---- test_all_local_index stdout ----
thread 'test_all_local_index' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:40221` failed, retrying 1/10
Failed to connect to `127.0.0.1:40221` failed, retrying 2/10
Failed to connect to `127.0.0.1:40221` failed, retrying 3/10
Failed to connect to `127.0.0.1:40221` failed, retrying 4/10
Failed to connect to `127.0.0.1:40221` failed, retrying 5/10
Failed to connect to `127.0.0.1:40221` failed, retrying 6/10
Failed to connect to `127.0.0.1:40221` failed, retrying 7/10
Failed to connect to `127.0.0.1:40221` failed, retrying 8/10
Failed to connect to `127.0.0.1:40221` failed, retrying 9/10
Failed to connect to `127.0.0.1:40221` failed, retrying 10/10
thread 'test_all_local_index' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:40221`', quickwit-cli/tests/cli.rs:891:35

---- test_ingest_docs_cli stdout ----
thread 'test_ingest_docs_cli' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:44289` failed, retrying 1/10
Failed to connect to `127.0.0.1:44289` failed, retrying 2/10
Failed to connect to `127.0.0.1:44289` failed, retrying 3/10
Failed to connect to `127.0.0.1:44289` failed, retrying 4/10
Failed to connect to `127.0.0.1:44289` failed, retrying 5/10
Failed to connect to `127.0.0.1:44289` failed, retrying 6/10
Failed to connect to `127.0.0.1:44289` failed, retrying 7/10
Failed to connect to `127.0.0.1:44289` failed, retrying 8/10
Failed to connect to `127.0.0.1:44289` failed, retrying 9/10
Failed to connect to `127.0.0.1:44289` failed, retrying 10/10
thread 'test_ingest_docs_cli' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:44289`', quickwit-cli/tests/cli.rs:236:35

---- test_garbage_collect_index_cli stdout ----
thread 'test_garbage_collect_index_cli' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:34349` failed, retrying 1/10
Failed to connect to `127.0.0.1:34349` failed, retrying 2/10
Failed to connect to `127.0.0.1:34349` failed, retrying 3/10
Failed to connect to `127.0.0.1:34349` failed, retrying 4/10
Failed to connect to `127.0.0.1:34349` failed, retrying 5/10
Failed to connect to `127.0.0.1:34349` failed, retrying 6/10
Failed to connect to `127.0.0.1:34349` failed, retrying 7/10
Failed to connect to `127.0.0.1:34349` failed, retrying 8/10
Failed to connect to `127.0.0.1:34349` failed, retrying 9/10
Failed to connect to `127.0.0.1:34349` failed, retrying 10/10
thread 'test_garbage_collect_index_cli' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:34349`', quickwit-cli/tests/cli.rs:748:35

---- test_garbage_collect_cli_no_grace stdout ----
thread 'test_garbage_collect_cli_no_grace' panicked at 'The telemetry loop is already started.', quickwit-telemetry/src/sender.rs:270:9
Failed to connect to `127.0.0.1:36301` failed, retrying 1/10
Failed to connect to `127.0.0.1:36301` failed, retrying 2/10
Failed to connect to `127.0.0.1:36301` failed, retrying 3/10
Failed to connect to `127.0.0.1:36301` failed, retrying 4/10
Failed to connect to `127.0.0.1:36301` failed, retrying 5/10
Failed to connect to `127.0.0.1:36301` failed, retrying 6/10
Failed to connect to `127.0.0.1:36301` failed, retrying 7/10
Failed to connect to `127.0.0.1:36301` failed, retrying 8/10
Failed to connect to `127.0.0.1:36301` failed, retrying 9/10
Failed to connect to `127.0.0.1:36301` failed, retrying 10/10
thread 'test_garbage_collect_cli_no_grace' panicked at 'called `Result::unwrap()` on an `Err` value: too many attempts to connect to `127.0.0.1:36301`', quickwit-cli/tests/cli.rs:637:35


failures:
    test_all_local_index
    test_cmd_search_aggregation
    test_delete_index_cli
    test_garbage_collect_cli_no_grace
    test_garbage_collect_index_cli
    test_ingest_docs_cli

test result: FAILED. 10 passed; 6 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.82s

error: test failed, to rerun pass `-p quickwit-cli --test cli`

[fulmicoton]

It would be nice to fix this of course, but could disable telemetry in your env variables? We should also disable it on debug mode probably?

[PSeitz]

It's still flaky with telemetry disabled

thread 'test_ingest_docs_cli' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `1`', quickwit-cli/tests/cli.rs:261:5


failures:
    test_ingest_docs_cli

test result: FAILED. 15 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 3.14s

error: test failed, to rerun pass `-p quickwit-cli --test cli`