dhat-rs
dhat-rs copied to clipboard
Published
20 hours ago •
nnethercote
nnethercote
Panic in `record_block`
Using dhat==0.3.0, I see a fairly reproducible (although non-deterministic) panic in record_block. Some smaller runs complete successfully, while larger runs generally encounter it before they can complete.
The Profiler is constructed via:
dhat::Profiler::builder()
.file_name(path)
.build()
thread 'tokio-runtime-worker' panicked at 'assertion failed: matches!(old, None)', /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/dhat-0.3.0/src/lib.rs:538:9
stack backtrace:
0: rust_begin_unwind
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:498:5
1: core::panicking::panic_fmt
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:107:14
2: core::panicking::panic
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panicking.rs:48:5
3: dhat::Globals::record_block
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/dhat-0.3.0/src/lib.rs:538:9
4: <dhat::Alloc as core::alloc::global::GlobalAlloc>::alloc
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/dhat-0.3.0/src/lib.rs:1230:17
5: __rg_alloc
at /Users/stuhood/src/pants/src/rust/engine/src/lib.rs:62:1
6: alloc::alloc::alloc
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/alloc.rs:87:14
7: alloc::alloc::Global::alloc_impl
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/alloc.rs:169:73
8: <alloc::alloc::Global as core::alloc::Allocator>::allocate
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/alloc.rs:229:9
9: alloc::raw_vec::RawVec<T,A>::allocate_in
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/raw_vec.rs:185:45
10: alloc::raw_vec::RawVec<T,A>::with_capacity_in
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/raw_vec.rs:132:9
11: alloc::vec::Vec<T,A>::with_capacity_in
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/vec/mod.rs:609:20
12: <T as alloc::slice::hack::ConvertVec>::to_vec
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/slice.rs:204:27
13: alloc::slice::hack::to_vec
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/slice.rs:176:9
14: alloc::slice::<impl [T]>::to_vec_in
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/slice.rs:498:9
15: <alloc::vec::Vec<T,A> as core::clone::Clone>::clone
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/vec/mod.rs:2480:9
16: <glob::Pattern as core::clone::Clone>::clone
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/glob-0.3.0/src/lib.rs:486:5
17: <T as alloc::slice::hack::ConvertVec>::to_vec
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/slice.rs:211:32
18: alloc::slice::hack::to_vec
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/slice.rs:176:9
19: alloc::slice::<impl [T]>::to_vec_in
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/slice.rs:498:9
20: alloc::slice::<impl [T]>::to_vec
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/slice.rs:474:9
21: fs::glob_matching::PathGlob::parse_globs
at /Users/stuhood/src/pants/src/rust/engine/fs/src/glob_matching.rs:236:9
22: fs::glob_matching::PathGlob::parse
at /Users/stuhood/src/pants/src/rust/engine/fs/src/glob_matching.rs:157:5
23: fs::glob_matching::PathGlob::spread_filespecs
at /Users/stuhood/src/pants/src/rust/engine/fs/src/glob_matching.rs:101:19
24: fs::glob_matching::PreparedPathGlobs::create
at /Users/stuhood/src/pants/src/rust/engine/fs/src/glob_matching.rs:301:19
25: fs::PathGlobs::parse
at /Users/stuhood/src/pants/src/rust/engine/fs/src/lib.rs:425:5
26: <engine::nodes::Snapshot as engine::nodes::WrappedNode>::run_wrapped_node::__run_wrapped_node::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/src/nodes.rs:874:22
27: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
28: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/future.rs:119:9
29: <F as futures_core::future::TryFuture>::try_poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.21/src/future.rs:82:9
30: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/future/try_future/into_future.rs:34:9
31: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/future/future/map.rs:55:37
32: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/lib.rs:91:13
33: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/lib.rs:91:13
34: <engine::nodes::NodeKey as graph::node::Node>::run::__run::{{closure}}::{{closure}}::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/src/nodes.rs:1467:13
35: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
36: <engine::nodes::NodeKey as graph::node::Node>::run::__run::{{closure}}::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/workunit_store/src/lib.rs:855:24
37: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
38: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:28
39: tokio::task::task_local::TaskLocalFuture<T,F>::with_task
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:272:9
40: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:9
41: workunit_store::scope_task_workunit_store_handle::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/workunit_store/src/lib.rs:960:3
42: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
43: <engine::nodes::NodeKey as graph::node::Node>::run::__run::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/src/nodes.rs:1412:5
44: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
45: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/future.rs:119:9
46: graph::entry::Entry<N>::spawn_node_execution::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/graph/src/entry.rs:347:19
47: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
48: graph::entry::Entry<N>::spawn_node_execution::{{closure}}::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/macros/select.rs:505:49
49: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/future/poll_fn.rs:38:9
50: graph::entry::Entry<N>::spawn_node_execution::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/graph/src/entry.rs:354:23
51: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
52: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:28
53: tokio::task::task_local::TaskLocalFuture<T,F>::with_task
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:272:9
54: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:9
55: workunit_store::scope_task_workunit_store_handle::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/workunit_store/src/lib.rs:960:3
56: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
57: task_executor::Executor::future_with_correct_context::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/task_executor/src/lib.rs:184:7
58: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
59: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:28
60: tokio::task::task_local::TaskLocalFuture<T,F>::with_task
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:272:9
61: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/task/task_local.rs:280:9
62: stdio::scope_task_destination::{{closure}}
at /Users/stuhood/src/pants/src/rust/engine/stdio/src/lib.rs:466:3
63: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/future/mod.rs:80:19
64: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/core.rs:161:17
65: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/loom/std/unsafe_cell.rs:14:9
66: tokio::runtime::task::core::CoreStage<T>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/core.rs:151:13
67: tokio::runtime::task::harness::poll_future::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:461:19
68: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/panic/unwind_safe.rs:271:9
69: std::panicking::try::do_call
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:406:40
70: <unknown>
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:434:6
71: std::panicking::try
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panicking.rs:370:19
72: std::panic::catch_unwind
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/panic.rs:133:14
73: tokio::runtime::task::harness::poll_future
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:449:18
74: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:98:27
75: tokio::runtime::task::harness::Harness<T,S>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/harness.rs:53:15
76: tokio::runtime::task::raw::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/raw.rs:113:5
77: tokio::runtime::task::raw::RawTask::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/raw.rs:70:18
78: tokio::runtime::task::LocalNotified<S>::run
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/mod.rs:343:9
79: tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:448:21
80: tokio::coop::with_budget::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:102:9
81: std::thread::local::LocalKey<T>::try_with
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/thread/local.rs:399:16
82: std::thread::local::LocalKey<T>::with
at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/std/src/thread/local.rs:375:9
83: tokio::coop::with_budget
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:95:5
84: tokio::coop::budget
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:72:5
85: tokio::runtime::thread_pool::worker::Context::run_task
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:424:9
86: tokio::runtime::thread_pool::worker::Context::run
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:391:24
87: tokio::runtime::thread_pool::worker::run::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:376:17
88: tokio::macros::scoped_tls::ScopedKey<T>::set
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/macros/scoped_tls.rs:61:9
89: tokio::runtime::thread_pool::worker::run
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:373:5
90: tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/thread_pool/worker.rs:352:45
91: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/blocking/task.rs:42:21
92: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/task/core.rs:161:17
93: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /Users/stuhood/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/loom/std/unsafe_cell.rs:14:9
94: tokio::runtime::task::core::CoreStage<T>::poll
See https://github.com/pantsbuild/pants/compare/main...stuhood:stuhood/dhat for the relevant diff enabling the tool.
Thanks for the report. Do you have steps to reproduce? I just tried MODE=debug RUST_BACKTRACE=1 ./pants --no-pantsd help on my Mac, because that's what triggered the problem in #15, but that ran successfully.
I also tried ./pants test :: and got this:
$ ./pants test ::
10:24:05.19 [INFO] Initializing scheduler...
10:24:05.54 [INFO] Scheduler initialized.
⠉ 13.02s `test` goal
⠁
⠁
⠁
⠁
⠁
⠁
⠁
⠁
⠁
⠁
⠁
⠁
⠁
⠁
⠁
Traceback (most recent call last):
File "/Users/njn/dev/pants/src/python/pants/bin/pants_loader.py", line 119, in <module>
main()
File "/Users/njn/dev/pants/src/python/pants/bin/pants_loader.py", line 115, in main
PantsLoader.main()
File "/Users/njn/dev/pants/src/python/pants/bin/pants_loader.py", line 111, in main
cls.run_default_entrypoint()
File "/Users/njn/dev/pants/src/python/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
exit_code = runner.run(start_time)
File "/Users/njn/dev/pants/src/python/pants/bin/pants_runner.py", line 89, in run
return remote_runner.run(start_time)
File "/Users/njn/dev/pants/src/python/pants/bin/remote_pants_runner.py", line 117, in run
return self._connect_and_execute(pantsd_handle, start_time)
File "/Users/njn/dev/pants/src/python/pants/bin/remote_pants_runner.py", line 151, in _connect_and_execute
return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.
If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). You can set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`). You can also disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.
If neither of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (https://www.pantsbuild.org/docs/getting-help).
@mnethercote I can reproduce this every time in a repo I'm working on.
thread 'tokio-runtime-worker' panicked at 'assertion failed: matches!(old, None)', /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/dhat-0.3.0/src/lib.rs:538:9
stack backtrace:
0: rust_begin_unwind
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
2: core::panicking::panic
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:48:5
3: dhat::Globals::record_block
at /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/dhat-0.3.0/src/lib.rs:538:9
4: <dhat::Alloc as core::alloc::global::GlobalAlloc>::alloc
at /Users/evan/.cargo/registry/src/github.com-1ecc6299db9ec823/dhat-0.3.0/src/lib.rs:1230:17
5: __rg_alloc
at /Users/evan/src/m/narwhal/node/src/main.rs:32:15
6: alloc::alloc::alloc
at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/alloc/src/alloc.rs:87:14
Branch/repo:
To github.com:MystenLabs/narwhal.git
- [new branch] ec/memory-profiling -> ec/memory-profiling
If you follow the quick start (https://github.com/MystenLabs/narwhal#quick-start) install the Python deps and cd into benchmark/ and run fab local you should be able to reproduce it -
The output from running will say:
ERROR: Failed to run benchmark
Caused by:
0: <class 'benchmark.logs.ParseError'>
1: Primary(s) panicked
and if you look in logs/primary-*.log within benchmark folder you should find the panic and stack trace in one of the files.
I was hitting a problem with panics in record_block() and also sometimes arbitrary hangs when my program started.
Environment:
- Hardware: MacBook Pro (2021, M1 Max)
- OS: macos: 12.6
- Toolchain: rust 1.63.0
I cloned your repo, replaced parking_lot Mutex with std Mutex and no more problems. The performance seems acceptable for my use case.
Would you be open to a patch which made this change on macos? Or, more specifically M1 macos? I'm happy to do that if you are interested.
@stuhood: Can you try the new 0.3.2 release? It has a bugfix that might solve your problem. Thanks.
@stuhood: Can you try the new 0.3.2 release? It has a bugfix that might solve your problem. Thanks.
I made a few attempts, and no longer repro: thanks a lot @nnethercote, and sorry for disappearing in March.
I also tried
./pants test ::and got this:
This was likely a repro, but because pantsd was enabled (--no-pantsd wasn't set) the panic was occurring in the daemon process.