Overflowing multiplicities cause a crash

Open ggevay opened this issue 2 years ago • 6 comments

What version of Materialize are you using?

ac9a95501d1e9cc961174ac9a445d7d967840c67

How did you install Materialize?

Built from source

What is the issue?

The i64 Diffs quickly overflow in the following query, causing a crash:

WITH MUTUALLY RECURSIVE
  cnt (cnt int) AS (
    SELECT 1 AS cnt
    UNION ALL
    SELECT cnt+1 FROM ((SELECT * FROM cnt) UNION ALL (SELECT * FROM cnt)))
SELECT * FROM cnt;

It would be better to report a query-level error.

I guess this doesn't happen so often outside of WMR, but it can happen easily if one uses UNION ALL instead of UNION by mistake in a non-linear recursion, causing an exponential increase in the diffs.

Relevant log output

thread 'timely:work-0' panicked at 'attempt to add with overflow', /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/arith.rs:763:51
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
   2: core::panicking::panic
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:114:5
   3: <i64 as core::ops::arith::AddAssign>::add_assign
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/arith.rs:763:51
   4: <i64 as core::ops::arith::AddAssign<&i64>>::add_assign
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/internal_macros.rs:127:17
   5: <i64 as differential_dataflow::difference::Semigroup>::plus_equals
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/difference.rs:67:63
   6: differential_dataflow::consolidation::consolidate_updates_slice
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/consolidation.rs:125:17
   7: differential_dataflow::consolidation::consolidate_updates_from
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/consolidation.rs:95:18
   8: differential_dataflow::consolidation::consolidate_updates
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/consolidation.rs:86:5
   9: differential_dataflow::trace::implementations::merge_batcher::MergeSorter<D,T,R>::push
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/trace/implementations/merge_batcher.rs:233:13
  10: <differential_dataflow::trace::implementations::merge_batcher::MergeBatcher<B> as differential_dataflow::trace::Batcher<B>>::push_batch
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/trace/implementations/merge_batcher.rs:48:17
  11: <differential_dataflow::trace::rc_blanket_impls::RcBatcher<B> as differential_dataflow::trace::Batcher<alloc::rc::Rc<B>>>::push_batch
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/trace/mod.rs:420:93
  12: <differential_dataflow::collection::Collection<G,(K,V),R> as differential_dataflow::operators::arrange::arrangement::Arrange<G,K,V,R>>::arrange_core::{{closure}}::{{closure}}::{{closure}}
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/operators/arrange/arrangement.rs:590:25
  13: timely::dataflow::operators::generic::handles::InputHandleCore<T,D,P>::for_each
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/dataflow/operators/generic/handles.rs:94:13
  14: timely::dataflow::operators::generic::handles::FrontieredInputHandleCore<T,D,P>::for_each
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/dataflow/operators/generic/handles.rs:139:9
  15: <differential_dataflow::collection::Collection<G,(K,V),R> as differential_dataflow::operators::arrange::arrangement::Arrange<G,K,V,R>>::arrange_core::{{closure}}::{{closure}}
             at /home/gabor/.cargo/git/checkouts/differential-dataflow-d065d23d797aa027/cc88469/src/operators/arrange/arrangement.rs:588:21
  16: <timely::dataflow::stream::StreamCore<G,D1> as timely::dataflow::operators::generic::operator::Operator<G,D1>>::unary_frontier::{{closure}}::{{closure}}
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/dataflow/operators/generic/operator.rs:351:17
  17: timely::dataflow::operators::generic::builder_rc::OperatorBuilder<G>::build::{{closure}}::{{closure}}
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/dataflow/operators/generic/builder_rc.rs:133:31
  18: timely::dataflow::operators::generic::builder_rc::OperatorBuilder<G>::build_reschedule::{{closure}}
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/dataflow/operators/generic/builder_rc.rs:172:26
  19: <timely::dataflow::operators::generic::builder_raw::OperatorCore<T,L> as timely::scheduling::Schedule>::schedule
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/dataflow/operators/generic/builder_raw.rs:204:9
  20: timely::progress::subgraph::PerOperatorState<T>::schedule
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/progress/subgraph.rs:699:30
  21: timely::progress::subgraph::Subgraph<TOuter,TInner>::activate_child
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/progress/subgraph.rs:346:26
  22: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/progress/subgraph.rs:312:17
  23: timely::progress::subgraph::PerOperatorState<T>::schedule
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/progress/subgraph.rs:699:30
  24: timely::progress::subgraph::Subgraph<TOuter,TInner>::activate_child
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/progress/subgraph.rs:346:26
  25: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/progress/subgraph.rs:312:17
  26: timely::worker::Wrapper::step::{{closure}}
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/worker.rs:754:57
  27: core::option::Option<T>::map
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/option.rs:970:29
  28: timely::worker::Wrapper::step
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/worker.rs:754:26
  29: timely::worker::Worker<A>::step_or_park
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/worker.rs:396:38
  30: mz_compute::server::Worker<A>::run_client
             at ./src/compute/src/server.rs:347:13
  31: mz_compute::server::Worker<A>::setup_channel_and_run_client
             at ./src/compute/src/server.rs:321:9
  32: mz_compute::server::Worker<A>::run
             at ./src/compute/src/server.rs:181:21
  33: <mz_compute::server::Config as mz_cluster::types::AsRunnableWorker<mz_compute_client::protocol::command::ComputeCommand,mz_compute_client::protocol::response::ComputeResponse>>::build_and_run
             at ./src/compute/src/server.rs:162:9
  34: mz_cluster::server::ClusterClient<mz_service::client::Partitioned<mz_service::local::LocalClient<C,R,<Worker as mz_cluster::types::AsRunnableWorker<C,R>>::Activatable>,C,R>,Worker,C,R>::build_timely::{{closure}}::{{closure}}
             at ./src/cluster/src/server.rs:179:13
  35: timely::execute::execute_from::{{closure}}
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/timely/src/execute.rs:388:22
  36: timely_communication::initialize::initialize_from::{{closure}}
             at /home/gabor/.cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/134842a/communication/src/initialize.rs:316:33


ci-regexp: attempt to add with overflow.*src/ops/arith.rs

Apr 12 '23 15:04 ggevay

This is unfortunately "works as designed." We don't have any alternative way to handle overflows in the diff. Also, this error should disappear in release mode and simply wrap-around.

Apr 13 '23 14:04 antiguru

Also, this error should disappear in release mode and simply wrap-around.

Then the query could give wrong results, right? (Maybe not this particular one, but some similar query.) Is that ok?

Apr 13 '23 14:04 ggevay

Another repro for this issue is reported here: https://github.com/MaterializeInc/materialize/issues/18645#issuecomment-1722977609

Sep 18 '23 11:09 vmarcos

And another repro is in https://github.com/MaterializeInc/materialize/issues/22234.

Oct 06 '23 15:10 vmarcos

Also, this error should disappear in release mode and simply wrap-around.

With ASan and release mode this panic is also occurring: https://buildkite.com/materialize/tests/builds/74647#018d6983-9098-4560-9dc2-583a5f307716

thread 'timely:work-1' panicked at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/ops/arith.rs:760:1:
attempt to add with overflow
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: <mz_compute::render::reduce::Accum as differential_dataflow::difference::Semigroup>::plus_equals
   4: <alloc::vec::Vec<mz_compute::render::reduce::Accum> as differential_dataflow::difference::Semigroup>::plus_equals
   5: differential_dataflow::consolidation::consolidate_updates_from::<(mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>
   6: <differential_dataflow::trace::implementations::merge_batcher_col::MergeSorterColumnation<(mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::finish_into
   7: <differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::trace::Batcher>::seal::<differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>
   8: <timely::dataflow::operators::generic::builder_raw::OperatorCore<mz_repr::timestamp::Timestamp, <timely::dataflow::operators::generic::builder_rc::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build_reschedule<<timely::dataflow::operators::generic::builder_rc::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build<<timely::dataflow::stream::StreamCore<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>> as timely::dataflow::operators::generic::operator::Operator<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>::unary_frontier<alloc::vec::Vec<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}::{closure#0}, timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>>::{closure#0}, <timely::dataflow::stream::StreamCore<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>> as timely::dataflow::operators::generic::operator::Operator<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>::unary_frontier<alloc::vec::Vec<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}::{closure#0}, timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>>::{closure#0}::{closure#0}>::{closure#0}, <timely::dataflow::operators::generic::builder_rc::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build<<timely::dataflow::stream::StreamCore<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>> as timely::dataflow::operators::generic::operator::Operator<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>::unary_frontier<alloc::vec::Vec<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}::{closure#0}, timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>>::{closure#0}, <timely::dataflow::stream::StreamCore<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>> as timely::dataflow::operators::generic::operator::Operator<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>::unary_frontier<alloc::vec::Vec<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}, <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_core<timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>, differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}::{closure#0}, timely::dataflow::channels::pact::ExchangeCore<alloc::vec::Vec<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>, ((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)), <differential_dataflow::collection::Collection<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, (mz_repr::row::Row, ()), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)> as differential_dataflow::operators::arrange::arrangement::Arrange<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::row::Row, (), (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>>::arrange_named<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBatch<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, (), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64)>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::key_batch::OrdKeyBuilder<mz_compute::row_spine::spines::RowLayout<((mz_repr::row::Row, ()), mz_repr::timestamp::Timestamp, (alloc::vec::Vec<mz_compute::render::reduce::Accum>, i64))>>>>>::{closure#0}>>::{closure#0}::{closure#0}>::{closure#0}::{closure#0}>::{closure#0}> as timely::scheduling::Schedule>::schedule
   9: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
  10: <timely::progress::subgraph::Subgraph<mz_repr::timestamp::Timestamp, mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
  11: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
  12: <timely::progress::subgraph::Subgraph<mz_repr::timestamp::Timestamp, mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
  13: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
  14: <timely::progress::subgraph::Subgraph<(), mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
  15: <timely::worker::Wrapper>::step
  16: <timely::worker::Worker<timely_communication::allocator::generic::Generic>>::step_or_park
  17: <mz_compute::server::Worker<timely_communication::allocator::generic::Generic>>::run_client
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Feb 05 '24 17:02 def-

Here is another instance in mz_join_core:

thread 'timely:work-1' panicked at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/ops/arith.rs:346:1:
attempt to multiply with overflow
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/panicking.rs:127:5
   3: <i64 as core::ops::arith::Mul>::mul
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/ops/arith.rs:346:1
   4: <i64 as core::ops::arith::Mul<&i64>>::mul
             at ./rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/internal_macros.rs:45:17
   5: <i64 as differential_dataflow::difference::Multiply>::multiply
             at ./cargo/git/checkouts/differential-dataflow-fbfe6a4d66ec22fc/15b7b8d/src/difference.rs:77:53
   6: <mz_compute::render::join::mz_join_core::Deferred<timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, differential_dataflow::trace::cursor::cursor_list::CursorList<differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>, differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, mz_repr::row::Row>>::work::<&mut <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, mz_compute::render::join::mz_join_core::mz_join_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, <mz_compute::render::join::linear_join::LinearJoinSpec>::render<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>>::{closure#0}>::{closure#0}::{closure#2}::{closure#5}>::{closure#1}::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0a64d05c687e98f0e-1/materialize/tests/src/compute/src/render/join/mz_join_core.rs:633:48
   7: <differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>> as differential_dataflow::trace::cursor::Cursor>::map_times::<<mz_compute::render::join::mz_join_core::Deferred<timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, differential_dataflow::trace::cursor::cursor_list::CursorList<differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>, differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, mz_repr::row::Row>>::work<&mut <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, mz_compute::render::join::mz_join_core::mz_join_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, <mz_compute::render::join::linear_join::LinearJoinSpec>::render<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>>::{closure#0}>::{closure#0}::{closure#2}::{closure#5}>::{closure#1}::{closure#0}>
             at ./cargo/git/checkouts/differential-dataflow-fbfe6a4d66ec22fc/15b7b8d/src/trace/implementations/ord_neu.rs:462:17
   8: <differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>> as differential_dataflow::trace::cursor::Cursor>::map_times::<<mz_compute::render::join::mz_join_core::Deferred<timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, differential_dataflow::trace::cursor::cursor_list::CursorList<differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>, differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, mz_repr::row::Row>>::work<&mut <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, mz_compute::render::join::mz_join_core::mz_join_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, <mz_compute::render::join::linear_join::LinearJoinSpec>::render<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>>::{closure#0}>::{closure#0}::{closure#2}::{closure#5}>::{closure#1}::{closure#0}>
             at ./cargo/git/checkouts/differential-dataflow-fbfe6a4d66ec22fc/15b7b8d/src/trace/mod.rs:438:13
   9: <mz_compute::render::join::mz_join_core::Deferred<timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, differential_dataflow::trace::cursor::cursor_list::CursorList<differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>, differential_dataflow::trace::rc_blanket_impls::RcBatchCursor<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValCursor<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, mz_repr::row::Row>>::work::<&mut <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, mz_compute::render::join::mz_join_core::mz_join_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>, <mz_compute::render::join::linear_join::LinearJoinSpec>::render<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, <mz_compute::render::context::Context<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>>>::differential_join_inner<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>, differential_dataflow::operators::arrange::agent::TraceAgent<differential_dataflow::trace::implementations::spine_fueled::Spine<alloc::rc::Rc<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBatch<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>, differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<mz_repr::row::Row, mz_repr::row::Row, timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64>, differential_dataflow::trace::rc_blanket_impls::RcBuilder<differential_dataflow::trace::implementations::ord_neu::val_batch::OrdValBuilder<mz_compute::row_spine::spines::RowRowLayout<((mz_repr::row::Row, mz_repr::row::Row), timely::order::product::Product<mz_repr::timestamp::Timestamp, differential_dataflow::dynamic::pointstamp::PointStamp<u64>>, i64)>>>>>>::{closure#2}, core::option::Option<mz_repr::row::Row>>::{closure#0}>::{closure#0}::{closure#2}::{closure#5}>::{closure#1}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-aarch64-585fc7f-i-0a64d05c687e98f0e-1/materialize/tests/src/compute/src/render/join/mz_join_core.rs:631:33

Feb 15 '24 16:02 teskje

Another one in https://buildkite.com/materialize/tests/builds/77062#018df3dc-85d8-46cf-b5f7-aab307423506

thread 'timely:work-1' panicked at src/compute/src/render/reduce.rs:2018:17:
attempt to add with overflow
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic
   3: <mz_compute::render::reduce::Accum as differential_dataflow::difference::Semigroup>::plus_equals
   4: differential_dataflow::difference::vector::<impl differential_dataflow::difference::Semigroup for alloc::vec::Vec<R>>::plus_equals
   5: differential_dataflow::consolidation::consolidate_updates_from
   6: differential_dataflow::trace::implementations::merge_batcher_col::MergeSorterColumnation<D,T,R>::finish_into
   7: <differential_dataflow::trace::implementations::merge_batcher_col::ColumnatedMergeBatcher<K,V,T,D> as differential_dataflow::trace::Batcher>::seal
   8: <timely::dataflow::operators::generic::builder_raw::OperatorCore<T,L> as timely::scheduling::Schedule>::schedule
   9: timely::progress::subgraph::PerOperatorState<T>::schedule
  10: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
  11: timely::progress::subgraph::PerOperatorState<T>::schedule
  12: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
  13: timely::progress::subgraph::PerOperatorState<T>::schedule
  14: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
  15: timely::worker::Wrapper::step
  16: timely::worker::Worker<A>::step_or_park
  17: mz_compute::server::Worker<A>::run_client
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Feb 29 '24 10:02 def-

Sentry Issue: DATABASE-BACKEND-3BF

Linking this issue, because it contains the same error as #23769, which is marked as duplicate it. This occurred in a production env.

Mar 14 '24 13:03 sentry[bot]