tikv
test_unretryable_failure failed when "make test"
Bug Report
What version of TiKV are you using?
commit 0b97a39d520afcd82a02d0f5c0966d6ccab2cd74 (latest master branch)
What operating system and CPU are you using?
rust toolchain: nightly-2023-12-28-aarch64-apple-darwin macos 14.1.1 (23B81), Apple M1 Pro
Steps to reproduce
just run "make test" in zsh
What did you expect?
i want to fix the bug by myself, and learn rust
What did happened?
~/tikv (master) » make test gjt@TIEMUHUAGUO-MB1
./scripts/test-all -- --nocapture
make[1]: Entering directory '/Users/gjt/tikv'
memory-engine pprof-fp jemalloc test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored
Finished test [unoptimized] target(s) in 2.09s
warning: the following packages contain code that will be rejected by a future version of Rust: criterion v0.3.5, fs_extra v1.1.0, nom v5.1.0, quick-xml v0.22.0
note: to see what the problems were, use the option --future-incompat-report, or run cargo report future-incompatibilities --id 2
Running unittests src/lib.rs (target/debug/deps/api_version-af3ca4bb13787175)
running 19 tests test keyspace::tests::test_v1_parse_keyspace ... ok test api_v2::tests::test_encoded_logical_delete ... ok test api_v2::tests::test_key_split_ts ... ok test keyspace::tests::test_v2_parse_keyspace ... ok test tests::test_meta_api_v2 ... ok test tests::test_convert_raw_user_key_range ... ok test api_v2::tests::test_decode_ts_from ... ok test tests::test_parse ... ok test tests::test_no_meta ... ok test tests::test_parse_range ... ok test tests::test_raw_key ... ok test tests::test_value_decode_err ... ok test tests::test_ttl ... ok test tests::test_raw_value_convert ... ok test tests::test_raw_key_convert ... ok test tests::test_value_valid ... ok test api_v2::tests::test_key_encode_err ... ok test api_v2::tests::test_append_ts_on_encoded_bytes ... ok test api_v2::tests::test_key_decode_err ... ok
test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/lib.rs (target/debug/deps/aws-b8b005554b43d433)
running 8 tests test s3::tests::test_s3_get_content_md5 ... ok test s3::tests::test_url_of_backend ... ok test s3::tests::test_try_read_exact ... ok test s3::tests::test_s3_config ... ok test kms::tests::test_kms_wrong_key_id ... ok test kms::tests::test_aws_kms ... ok test s3::tests::test_s3_storage_with_virtual_host ... ok test s3::tests::test_s3_storage_multi_part ... ok
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
Running unittests src/lib.rs (target/debug/deps/azure-60078384f434e0e6)
running 5 tests test azblob::tests::test_azblob_config ... ok test azblob::tests::test_config_check ... ok test azblob::tests::test_url_of_backend ... ok test kms::tests::test_init_azure_kms ... ok test kms::tests::test_azure_kms ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
Running unittests src/lib.rs (target/debug/deps/backup-08d653ecd8b2e8be)
running 22 tests test endpoint::tests::test_backup_file_name ... ok test endpoint::tests::test_control_thread_pool_adjust_keep_tasks ... ok test endpoint::tests::test_s3_config ... ok test endpoint::tests::test_backup_raw_apiv2_causal_ts ... ok test endpoint::tests::test_backup_replica_read ... ok test endpoint::tests::test_cancel ... ok test softlimit::softlimit_test::test_cpu_limit ... ok test softlimit::softlimit_test::test_cpu_limit_remain ... ok test softlimit::softlimit_test::test_current_idle ... ok test endpoint::tests::test_scan_error ... ok test utils::tests::test_is_valid_raw_value ... ok test utils::tests::test_key_value_codec ... ok test softlimit::softlimit_test::test_limit ... ok test endpoint::tests::test_adjust_thread_pool_size ... ok test endpoint::tests::test_seek_ranges_2 ... ok test endpoint::tests::test_seek_range ... ok test endpoint::tests::test_seek_ranges ... ok test service::tests::test_client_stop ... ok test writer::tests::test_writer ... ok test endpoint::tests::test_handle_backup_task ... ok test endpoint::tests::test_handle_backup_raw ... ok test endpoint::tests::test_busy ... ok
test result: ok. 22 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.71s
Running unittests src/lib.rs (target/debug/deps/backup_stream-7c639a97b96e99da)
running 68 tests
test errors::test::baseline ... ok
test errors::test::contextual_add_format_strings ... ok
test errors::test::contextual_add_closure ... ok
test errors::test::contextual_add_format_strings_directly ... ok
test errors::test::contextual_ok ... ok
test errors::test::test_contextual_error ... ok
test checkpoint_manager::tests::test_mgr ... ok
test checkpoint_manager::tests::test_mgr_freeze ... ok
test checkpoint_manager::tests::test_flush ... ok
test metadata::client::test::test_parse ... ok
test checkpoint_manager::tests::test_rpc_sub ... ok
test checkpoint_manager::tests::test_after ... ok
test checkpoint_manager::tests::test_rpc_failure ... ok
test metadata::test::test_basic ... ok
test metadata::test::test_init ... ok
test metadata::test::test_storage_checkpoint_of ... ok
test metadata::test::test_set_storage_checkpoint ... ok
test metadata::test::test_progress ... ok
test metadata::test::test_watch ... ok
test observer::tests::test_observer_cancel ... ok
storage="/var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da"
test observer::tests::test_hibernate ... ok
storage="/var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/e2b2d433-c035-4478-8bb8-fba63f21e8b8"
test router::tests::test_cleanup_when_stop ... ok
test router::tests::test_decode_begin_ts ... ok
test observer::tests::test_observer_basic ... ok
test router::tests::test_empty_resolved_ts ... ok
test router::tests::test_est_len_in_flush ... ok
storage="/var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/d3e5e835-ce0f-4d45-8bae-a02d9432e38e"
test router::tests::test_flush_on_events_race ... ok
storage="/var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/81e963a5-625f-4c08-9590-5dbd2b8f47eb"
test router::tests::test_do_flush ... ok
storage="/var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/73a71466-3d40-4af2-b9ca-535c60bcded1"
test router::tests::test_flush_with_pausing_self ... ok
test router::tests::test_format_datetime ... ok
test router::tests::test_register ... ok
test router::tests::test_selector ... ok
test router::tests::test_udpate_invalid_config ... ok
test router::tests::test_update_config ... ok
test router::tests::test_update_global_checkpoint ... ok
test subscription_manager::test::test_always_failure_initial_scan ... ok
test subscription_manager::test::test_backoff_for_start_observe ... ok
test router::tests::test_flush_with_error ... ok
test metadata::checkpoint_cache::test::test_basic ... ok
test subscription_manager::test::test_basic_retry ... ok
test subscription_manager::test::test_on_high_mem ... ok
thread 'subscription_manager::test::test_unretryable_failure' panicked at components/backup-stream/src/subscription_manager.rs:1078:14:
called Result::unwrap() on an Err value: Io(Os { code: 17, kind: AlreadyExists, message: "File exists" })
stack backtrace:
test subscription_track::test::test_cal_checkpoint ... ok
test subscription_track::test::test_delay_remove ... ok
test subscription_track::test::test_two_phase_resolver ... ok
test tempfiles::test::test_compression ... ok
test subscription_manager::test::test_region_split_inflight ... ok
test tempfiles::test::test_panic_not_leaked ... ok
test tempfiles::test::test_not_leaked ... ok
test tempfiles::test::test_read_many_times ... ok
test tempfiles::test::test_read ... ok
test tempfiles::test::test_swapout ... ok
test tempfiles::test::test_write_many_times ... ok
test event_loader::tests::test_disk_read ... ok
1test for compression writer. try to write to local path, and read it back.2,74
test utils::test::test_compression_writer ... ok
test utils::test::test_range_functions ... ok
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da/v1
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da/v1/backupmeta
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da/v1/backupmeta/1720757851162-b8f8f05d-4bb6-4a8b-be07-bae43916defb.meta
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da/v1/20240712
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da/v1/20240712/04
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da/v1/20240712/04/1
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da/v1/20240712/04/1/451086346135011328-68f81ef8-fb72-404e-8d69-2c144f723920.log
walking /var/folders/xk/6sr8z2m17bb320b0r38x5h7r0000gn/T/11dc9e79-1320-4e10-abb8-205db45115da/v1/20240712/04/1/451086346135011328-7400fb4a-d27d-4be5-8954-022f8a57f221.log
test router::tests::test_basic_file ... ok
test utils::test::test_redact ... ok
test utils::test::test_segment_tree ... ok
test utils::test::test_files_reader ... ok
test utils::test::test_recorder ... ok
test tempfiles::test::test_various_encryption ... ok
0: 0x104534560 - std::backtrace_rs::backtrace::libunwind::trace::h63a26b7f922d48be
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: 0x104534560 - std::backtrace_rs::backtrace::trace_unsynchronized::hd965fd1079def065
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x104534560 - std::sys_common::backtrace::_print_fmt::h46aacd3db9ea9c80
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:68:5
3: 0x104534560 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h79701f7bfd42493b
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:44:22
4: 0x104556634 - core::fmt::rt::Argument::fmt::hb4568520b46d326a
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/fmt/rt.rs:142:9
5: 0x104556634 - core::fmt::write::h68d6fc01ae5c4452
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/fmt/mod.rs:1120:17
6: 0x104530e48 - std::io::Write::write_fmt::ha312077183afba78
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/io/mod.rs:1810:15
7: 0x104534394 - std::sys_common::backtrace::_print::h6a83aa82425f6209
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:47:5
8: 0x104534394 - std::sys_common::backtrace::print::h93d2f4c2c9309eca
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:34:9
9: 0x104535cfc - std::panicking::default_hook::{{closure}}::he5d3e5fda0e121fc
10: 0x104535a44 - std::panicking::default_hook::h5c4259bfa5dbe81c
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:292:9
11: 0x104536144 - std::panicking::rust_panic_with_hook::h3f5fa8b2def27f76
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:779:13
12: 0x104536038 - std::panicking::begin_panic_handler::{{closure}}::h65e896afea09d287
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:657:13
13: 0x1045349e4 - std::sys_common::backtrace::__rust_end_short_backtrace::hc7ce49dedf831ac7
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:171:18
14: 0x104535dc0 - rust_begin_unwind
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:645:5
15: 0x1045d3628 - core::panicking::panic_fmt::hf894f44550f56762
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/panicking.rs:72:14
16: 0x1045d3a20 - core::result::unwrap_failed::h1470ce536ebb41f0
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/result.rs:1649:5
17: 0x102a7a220 - core::result::Result<T,E>::unwrap::h404edb795054797a
18: 0x10283a78c - backup_stream::subscription_manager::test::Suite::new::hf44a64646d8d0fbe
19: 0x102b0e860 - backup_stream::subscription_manager::test::test_unretryable_failure::h8e155fe42658dec1
20: 0x10285bb9c - backup_stream::subscription_manager::test::test_unretryable_failure::{{closure}}::h0f0ef7920a387311
21: 0x10290f15c - core::ops::function::FnOnce::call_once::h3350b07d88f70d63
22: 0x10450e2e8 - core::ops::function::FnOnce::call_once::h231fada922778131
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/ops/function.rs:250:5
23: 0x10450e2e8 - test::__rust_begin_short_backtrace::hfc8f83461be24888
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/test/src/lib.rs:627:18
24: 0x10450d2cc - test::run_test_in_process::{{closure}}::h706d45f13684ae48
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/test/src/lib.rs:650:60
25: 0x10450d2cc - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h0c04158a97d5d7fc
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/panic/unwind_safe.rs:272:9
26: 0x10450d2cc - std::panicking::try::do_call::hff4cc4d3d0c4c36d
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:552:40
27: 0x10450d2cc - std::panicking::try::h1dcdff3fe533a377
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:516:19
28: 0x10450d2cc - std::panic::catch_unwind::h22fbcc0a04638226
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panic.rs:142:14
29: 0x10450d2cc - test::run_test_in_process::h02e987757f9ecf4d
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/test/src/lib.rs:650:27
30: 0x10450d2cc - test::run_test::{{closure}}::h43594285d3a512ba
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/test/src/lib.rs:573:43
31: 0x1044df670 - test::run_test::{{closure}}::hc378b1925e9a62fd
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/test/src/lib.rs:601:41
32: 0x1044df670 - std::sys_common::backtrace::rust_begin_short_backtrace::h8c541319b31dc915
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:155:18
33: 0x1044e3e8c - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h4b3e56524d1662e4
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/thread/mod.rs:529:17
34: 0x1044e3e8c - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h8822bf25d4b6d0ad
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/panic/unwind_safe.rs:272:9
35: 0x1044e3e8c - std::panicking::try::do_call::hb3a053d3c91c5907
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:552:40
36: 0x1044e3e8c - std::panicking::try::h94e64f94b73fda84
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:516:19
37: 0x1044e3e8c - std::panic::catch_unwind::h4b449330c8d693d3
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panic.rs:142:14
38: 0x1044e3e8c - std::thread::Builder::spawn_unchecked::{{closure}}::h8ef6cab35b213aa4
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/thread/mod.rs:528:30
39: 0x1044e3e8c - core::ops::function::FnOnce::call_once{{vtable.shim}}::he3c059cb06d63cc3
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/ops/function.rs:250:5
40: 0x10453ac5c - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h63a4910eee98e3aa
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2015:9
41: 0x10453ac5c - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h8d24a93e7fea474d
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2015:9
42: 0x10453ac5c - std::sys::unix::thread::Thread::new::thread_start::h08e633abca8932bb
at /rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys/unix/thread.rs:108:17
43: 0x188b59034 - __pthread_joiner_wake
test subscription_manager::test::test_unretryable_failure ... FAILED
test metadata::store::pd::tests::test_watch ... ok
test metadata::store::pd::tests::test_query ... ok
test utils::test::test_wait_group ... ok
test endpoint::test::test_start ... ok
test metadata::store::pd::tests::test_retry ... ok
test metadata::store::pd::tests::test_check_error ... ok
failures:
failures: subscription_manager::test::test_unretryable_failure
test result: FAILED. 67 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.07s
error: test failed, to rerun pass -p backup-stream --lib
make[1]: Leaving directory '/Users/gjt/tikv'
I also had a similar problem, same as https://github.com/tikv/tikv/issues/14594
I feel like no one cares about these issues.
/cc @YuJuncen Seems the failure is due to multiple backup-steam tests concurrently visit the same temp file.
