vault icon indicating copy to clipboard operation
vault copied to clipboard

Published 20 hours ago • verified publisher icon hashicorp

Add AWS Auth WIF docs

Open Zlaticanin opened this issue 9 months ago • 2 comments

Zlaticanin avatar May 15 '24 18:05 Zlaticanin

@aljoscha apparently this continues to happen in Sentry while all previous GitHub tickets have been closed. Therefore the QA team collectively decided that a new GitHub ticket is warranted.

philip-stoev avatar May 03 '23 13:05 philip-stoev

I looked at the latest panics. These happened for a source after it had been dropped (as confirmed by looking at mz_audit_events. Still thinking about what a good solution would be, but there's no real problem right now, we might panic and restart a couple of times but the system will eventually (rather fast) converge to a good state.

aljoscha avatar May 06 '23 02:05 aljoscha

Also seen in parallel-workload today locally while trying to repro https://github.com/MaterializeInc/materialize/issues/22717:

git checkout cdbb5b6756a82498f28aa68350bb448cf2c84dd7
bin/mzcompose --find parallel-workload run default --seed=1700062842 --threads=32 --runtime=3000 --complexity=ddl --scenario=regression

thread 'timely:work-2' panicked at 'u388: sba9f9d73-d3b7-4e1f-b241-265d6b2b20af cannot serve requested as_of Antichain { elements: [1700064412487] }: Since(Antichain { elements: [1700064439000] })', /var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/persist-client/src/operators/shard_source.rs:356:13
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:67:14
   2: mz_persist_client::operators::shard_source::shard_source_descs::<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}::{closure#4}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/persist-client/src/operators/shard_source.rs:356:13
   3: <core::result::Result<mz_persist_client::read::Subscribe<mz_storage_types::sources::SourceData, (), mz_repr::timestamp::Timestamp, i64>, mz_persist_client::internal::state::Since<mz_repr::timestamp::Timestamp>>>::unwrap_or_else::<mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}::{closure#4}>
             at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/result.rs:1428:23
   4: mz_persist_client::operators::shard_source::shard_source_descs::<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/persist-client/src/operators/shard_source.rs:355:32
   5: <core::pin::Pin<alloc::boxed::Box<mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}>> as core::future::future::Future>::poll
             at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/future/future.rs:125:9
   6: <mz_timely_util::builder_async::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build::<mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}, mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}>::{closure#0}::{closure#0}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/timely-util/src/builder_async.rs:527:32
   7: <timely::dataflow::operators::generic::builder_rc::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build_reschedule::<<mz_timely_util::builder_async::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build<mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}, mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}>::{closure#0}, <mz_timely_util::builder_async::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build<mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}, mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}>::{closure#0}::{closure#0}>::{closure#0}
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/dataflow/operators/generic/builder_rc.rs:172:26
   8: <timely::dataflow::operators::generic::builder_raw::OperatorCore<mz_repr::timestamp::Timestamp, <timely::dataflow::operators::generic::builder_rc::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build_reschedule<<mz_timely_util::builder_async::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build<mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}, mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}>::{closure#0}, <mz_timely_util::builder_async::OperatorBuilder<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>>::build<mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}, mz_persist_client::operators::shard_source::shard_source_descs<mz_storage_types::sources::SourceData, (), i64, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#3}, timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source_core<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#2}::{closure#0}, mz_storage_operators::persist_source::persist_source<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>, mz_repr::timestamp::Timestamp>>::{closure#0}::{closure#0}>::{closure#1}::{closure#0}>::{closure#0}::{closure#0}>::{closure#0}> as timely::scheduling::Schedule>::schedule
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/dataflow/operators/generic/builder_raw.rs:204:9
   9: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:699:30
  10: <timely::progress::subgraph::Subgraph<mz_repr::timestamp::Timestamp, mz_repr::timestamp::Timestamp>>::activate_child
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:346:26
  11: <timely::progress::subgraph::Subgraph<mz_repr::timestamp::Timestamp, mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:312:17
  12: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:699:30
  13: <timely::progress::subgraph::Subgraph<mz_repr::timestamp::Timestamp, mz_repr::timestamp::Timestamp>>::activate_child
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:346:26
  14: <timely::progress::subgraph::Subgraph<mz_repr::timestamp::Timestamp, mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:312:17
  15: <timely::progress::subgraph::PerOperatorState<mz_repr::timestamp::Timestamp>>::schedule
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:699:30
  16: <timely::progress::subgraph::Subgraph<(), mz_repr::timestamp::Timestamp>>::activate_child
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:346:26
  17: <timely::progress::subgraph::Subgraph<(), mz_repr::timestamp::Timestamp> as timely::scheduling::Schedule>::schedule
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/progress/subgraph.rs:312:17
  18: <timely::worker::Wrapper>::step::{closure#0}
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/worker.rs:754:57
  19: <core::option::Option<&mut alloc::boxed::Box<dyn timely::scheduling::Schedule>>>::map::<bool, <timely::worker::Wrapper>::step::{closure#0}>
             at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/option.rs:1075:29
  20: <timely::worker::Wrapper>::step
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/worker.rs:754:48
  21: <timely::worker::Worker<timely_communication::allocator::generic::Generic>>::step_or_park
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/worker.rs:396:38
  22: <mz_compute::server::Worker<timely_communication::allocator::generic::Generic>>::run_client
             at ./var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/compute/src/server.rs:390:13
  23: <mz_compute::server::Worker<timely_communication::allocator::generic::Generic>>::setup_channel_and_run_client
             at ./var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/compute/src/server.rs:363:9
  24: <mz_compute::server::Worker<timely_communication::allocator::generic::Generic>>::run
             at ./var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/compute/src/server.rs:227:21
  25: <mz_compute::server::Config as mz_cluster::types::AsRunnableWorker<mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse>>::build_and_run::<timely_communication::allocator::generic::Generic>
             at ./var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/compute/src/server.rs:207:9
  26: <mz_cluster::server::ClusterClient<mz_service::client::Partitioned<mz_service::local::LocalClient<mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse, timely::scheduling::activate::SyncActivator>, mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse>, mz_compute::server::Config, mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse>>::build_timely::{closure#0}::{closure#2}
             at ./var/lib/buildkite-agent/builds/buildkite-builders-d43b1b5-i-08b7e913dad28db18-1/materialize/tests/src/cluster/src/server.rs:235:13
  27: timely::execute::execute_from::<timely_communication::allocator::generic::GenericBuilder, (), <mz_cluster::server::ClusterClient<mz_service::client::Partitioned<mz_service::local::LocalClient<mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse, timely::scheduling::activate::SyncActivator>, mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse>, mz_compute::server::Config, mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse>>::build_timely::{closure#0}::{closure#2}>::{closure#0}
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/timely/src/execute.rs:388:22
  28: timely_communication::initialize::initialize_from::<timely_communication::allocator::generic::GenericBuilder, (), timely::execute::execute_from<timely_communication::allocator::generic::GenericBuilder, (), <mz_cluster::server::ClusterClient<mz_service::client::Partitioned<mz_service::local::LocalClient<mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse, timely::scheduling::activate::SyncActivator>, mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse>, mz_compute::server::Config, mz_compute_client::protocol::command::ComputeCommand, mz_compute_client::protocol::response::ComputeResponse>>::build_timely::{closure#0}::{closure#2}>::{closure#0}>::{closure#0}
             at ./cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/53adc68/communication/src/initialize.rs:316:33
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

def- avatar Nov 15 '23 16:11 def-

Seen again in Nightly https://buildkite.com/materialize/nightlies/builds/5547#018c4968-c0c9-4b4b-9b05-18b74983b415

philip-stoev avatar Dec 08 '23 14:12 philip-stoev

And again https://buildkite.com/materialize/nightlies/builds/6013

philip-stoev avatar Jan 18 '24 12:01 philip-stoev

This is still happening commonly in CI, see for example https://buildkite.com/materialize/tests/builds/77445#018e0a6e-3213-41df-966f-8387b87cc0eb

def- avatar Mar 06 '24 16:03 def-

I saw this again here: https://buildkite.com/materialize/tests/builds/78644#018e53a5-b742-4465-bc80-8cc1915fed78

originally looked like my code was broken, but im not so sure

guswynn avatar Mar 18 '24 23:03 guswynn

This panic is still happening a lot, 5 times in a single nightly run here for example: https://buildkite.com/materialize/nightlies/builds/7102

def- avatar Mar 27 '24 21:03 def-

See also the discussion in https://materializeinc.slack.com/archives/C01CFKM1QRF/p1712052300467749.

nrainer-materialize avatar Apr 05 '24 07:04 nrainer-materialize

Sentry Issue: DATABASE-BACKEND-3CT

sentry-io[bot] avatar Apr 19 '24 02:04 sentry-io[bot]

This still occurs. For example, in https://buildkite.com/materialize/nightly/builds/7721#018f5d1f-804b-42d9-a79d-44fbb06e6b1e.

nrainer-materialize avatar May 09 '24 14:05 nrainer-materialize

https://buildkite.com/materialize/nightly/builds/7741#annotation-018f676e-705b-433b-8170-03038caa8a14-error

nrainer-materialize avatar May 13 '24 08:05 nrainer-materialize