materialize icon indicating copy to clipboard operation
materialize copied to clipboard
verified publisher icon MaterializeInc

clusterd segfault: avro-decode-multi-record.td:30:1: error: preparing query failed: connection closed

Open def- opened this issue 1 year ago • 8 comments

What version of Materialize are you using?

726526ee19

What is the issue?

This was seen in https://buildkite.com/materialize/nightlies/builds/6135#018d3b86-94b6-4fb1-96a6-fab9027d5287 But is most likely unrelated to the PR and rather related to switching CI to aarch64.

I'm tentatively marking this as a release blocker as it scares me and we should figure this out before converting anyone to aarch64 in production.

> CREATE SOURCE avro_decode_type_multi_record
IN CLUSTER quickstart
FROM KAFKA CONNECTION kafka_conn (TOPIC 'testdrive-avro-decode-type-multi-record-4102834156')
FORMAT AVRO USING SCHEMA '[{"type": "record", "name": "f1", "fields": [ {"name": "f1", "type": "int"} ] }, {"type": "record", "name": "f2", "fields": [ {"name": "f2", "type": "int"} ] } ]'
ENVELOPE NONE
rows match; continuing at ts 1706102892.2559955
> SELECT (f1).f1, (f2).f2 FROM avro_decode_type_multi_record
rows didn't match; sleeping to see if dataflow catches up 50ms 75ms 113ms 169ms 253ms 380ms 570ms 854ms 1s 2s 3s 4s 6s 10s 15s 22s 11s
^^^ +++
avro-decode-multi-record.td:30:1: error: preparing query failed: connection closed
     |
  29 | 
  30 | > SELECT (f1).f1, (f2).f2 FROM avro_decode_type_multi_record
     | ^

environmentd went down because of crash propagation, the clusterd problem seems to have been the segfault recorded in buildkite artifacts: https://gist.github.com/def-/54201a7ee51a602ffc9ce58481d466cf Excerpt:

Core was generated by `/usr/local/bin/clusterd --storage-controller-listen-addr=/tmp/3b0cd41d1058093e8'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000ffff9638e470 in __pthread_mutex_clocklock_common (mutex=0xffff41e5a8c0, clockid=2066113312, abstime=0xffff7b2683b0) at ./nptl/pthread_mutex_timedlock.c:250
[Current thread is 1 (LWP 857)]
#0  0x0000ffff9638e470 in __pthread_mutex_clocklock_common (mutex=0xffff41e5a8c0, clockid=2066113312, abstime=0xffff7b2683b0) at ./nptl/pthread_mutex_timedlock.c:250
        err = 0
        assume_other_futex_waiters = 2147483648
        oldval = 1
        id = 0
        result = 0
        out = <optimized out>
        __PRETTY_FUNCTION__ = "\001\004\002\002\001\004\002\n\t\004\002\002\t\004\002\n\000\000\000\000\000\001\000\000\000\000\b\000\000\001\b\000"
#1  0x0000aaaabdce4c54 in core::ptr::drop_in_place<std::sys::unix::thread::Thread> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
No locals.
#2  core::ptr::drop_in_place<std::thread::JoinInner<()>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
No locals.
#3  core::ptr::drop_in_place<std::thread::JoinHandle<()>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
No locals.
#4  core::ptr::drop_in_place<mz_ore::thread::UnparkOnDropHandle<()>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
No locals.
#5  0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 295 (LWP 856):
#0  core::ptr::drop_in_place<mz_ore::thread::UnparkOnDropHandle<()>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#1  0x0000aaaabdecaa88 in core::ptr::drop_in_place<mz_storage::source::kafka::KafkaSourceReader> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#2  0x0000aaaabdc9ebc0 in core::ptr::drop_in_place<mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>> () at src/storage/src/source/kafka.rs:682
#3  core::ptr::drop_in_place<alloc::boxed::Box<mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>, alloc::alloc::Global>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#4  core::ptr::drop_in_place<core::pin::Pin<alloc::boxed::Box<mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>, alloc::alloc::Global>>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#5  core::ptr::drop_in_place<core::option::Option<core::pin::Pin<alloc::boxed::Box<mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>, alloc::alloc::Global>>>> () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ptr/mod.rs:497
#6  0x0000aaaabe5d6644 in mz_timely_util::builder_async::{impl#12}::build::{closure#0}::{closure#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::kafka::{impl#0}::render::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>, mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>> (new_frontiers=...) at src/timely-util/src/builder_async.rs:553
#7  timely::dataflow::operators::generic::builder_rc::{impl#0}::build_reschedule::{closure#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_timely_util::builder_async::{impl#12}::build::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::kafka::{impl#0}::render::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>, mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_timely_util::builder_async::{impl#12}::build::{closure#0}::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::kafka::{impl#0}::render::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>, mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>> (progress=<optimized out>) at /cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/de20aa8/timely/src/dataflow/operators/generic/builder_rc.rs:172
#8  timely::dataflow::operators::generic::builder_raw::{impl#2}::schedule<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, timely::dataflow::operators::generic::builder_rc::{impl#0}::build_reschedule::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_timely_util::builder_async::{impl#12}::build::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::kafka::{impl#0}::render::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>, mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_timely_util::builder_async::{impl#12}::build::{closure#0}::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::kafka::{impl#0}::render::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>, mz_storage::source::kafka::{impl#0}::render::{closure#0}::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, futures_util::stream::stream::Inspect<async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::source::source_reader_pipeline::source_render_operator::{closure_env#0}<timely::dataflow::scopes::child::Child<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>, async_stream::async_stream::AsyncStream<timely::progress::frontier::Antichain<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>, mz_storage::source::source_reader_pipeline::reclock_resume_upper::{async_block_env#0}<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>, mz_repr::timestamp::Timestamp>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>, mz_storage::render::sources::render_source::{async_block_env#0}<timely::dataflow::scopes::child::Child<timely::worker::Worker<timely_communication::allocator::generic::Generic>, ()>, mz_storage_types::sources::kafka::KafkaSourceConnection<mz_storage_types::connections::inline::InlinedConnection>>>>>> (self=<optimized out>) at /cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/de20aa8/timely/src/dataflow/operators/generic/builder_raw.rs:204
#9  0x0000aaaac049cca0 in timely::progress::subgraph::PerOperatorState<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>::schedule<mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>> (self=0xffff78cde898) at /cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/de20aa8/timely/src/progress/subgraph.rs:699
#10 timely::progress::subgraph::Subgraph<(), mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>>::activate_child<(), mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>> (self=0xffff90968700, child_index=1) at /cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/de20aa8/timely/src/progress/subgraph.rs:346
#11 timely::progress::subgraph::{impl#1}::schedule<(), mz_timely_util::order::Partitioned<mz_storage_types::sources::kafka::RangeBound<i32>, mz_storage_types::sources::MzOffset>> (self=0xffff90968700) at /cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/de20aa8/timely/src/progress/subgraph.rs:312
#12 0x0000aaaac04a67a4 in timely::progress::subgraph::PerOperatorState<()>::schedule<()> (self=0xffff78cde430) at /cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/de20aa8/timely/src/progress/subgraph.rs:699
#13 timely::progress::subgraph::Subgraph<(), ()>::activate_child<(), ()> (self=0xffff90967500, child_index=2) at /cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/de20aa8/timely/src/progress/subgraph.rs:346
#14 timely::progress::subgraph::{impl#1}::schedule<(), ()> (self=0xffff90967500) at /cargo/git/checkouts/timely-dataflow-70b80d81d6cabd62/de20aa8/timely/src/progress/subgraph.rs:312
#15 0x0000aaaac209d1f4 in timely::worker::{impl#7}::step::{closure#0} (op=<optimized out>) at src/worker.rs:754

So I'm guessing the kafka source is causing this segfault.

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

This indicates to me that there could be memory corruption as the root cause. For context: On aarch64 writes can be reordered relative to other writes, on x86-64 they can not. Example blog post about this: https://www.nickwilcox.com/blog/arm_vs_x86_memory_model/

def- avatar Jan 24 '24 14:01 def-

Removed release blocker but this should block the (probably non-existing) aarch64 migration epic.

def- avatar Jan 24 '24 14:01 def-

Another one: https://buildkite.com/materialize/tests/builds/74408#018d5cd0-22ac-49cb-99b3-19aaebb57063

def- avatar Jan 31 '24 11:01 def-

This is still happening after all the rdkafka fixes btw: https://buildkite.com/materialize/tests/builds/74739#018d7122-2cc0-4213-b9ba-44a51c56f7ed

def- avatar Feb 04 '24 12:02 def-

This one seems interesting: (via coverage run https://buildkite.com/materialize/coverage/builds/348#018d7e0a-92fb-4905-a5cb-28cdfd231aa9)

Program terminated with signal SIGSEGV, Segmentation fault.
#0  core::sync::atomic::atomic_compare_exchange<u32> (dst=0x0, old=0, new=1, success=core::sync::atomic::Ordering::Acquire, failure=core::sync::atomic::Ordering::Relaxed) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/sync/atomic.rs:3262
[Current thread is 1 (LWP 19365)]
#0  core::sync::atomic::atomic_compare_exchange<u32> (dst=0x0, old=0, new=1, success=core::sync::atomic::Ordering::Acquire, failure=core::sync::atomic::Ordering::Relaxed) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/sync/atomic.rs:3262
        val = <optimized out>
        ok = <optimized out>
#1  core::sync::atomic::AtomicU32::compare_exchange () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/sync/atomic.rs:2995
        failure = core::sync::atomic::Ordering::Relaxed
        success = core::sync::atomic::Ordering::Acquire
        new = 1
        current = 0
#2  std::sys::unix::locks::futex_mutex::Mutex::lock () at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys/unix/locks/futex_mutex.rs:27
        self = 0x0
#3  std::sync::mutex::Mutex<crossbeam_channel::flavors::zero::Inner>::lock<crossbeam_channel::flavors::zero::Inner> (self=0x0) at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sync/mutex.rs:273
No locals.
#4  crossbeam_channel::flavors::zero::Channel<mz_kafka_util::client::MzKafkaError>::send<mz_kafka_util::client::MzKafkaError> (self=0x0, msg=..., deadline=...) at /cargo/registry/src/index.crates.io-6f17d22bba15001f/crossbeam-channel-0.5.8/src/flavors/zero.rs:225
        token = 0xffff57418640
#5  0x0000aaaac9e99a80 in crossbeam_channel::channel::Sender<mz_kafka_util::client::MzKafkaError>::send<mz_kafka_util::client::MzKafkaError> (self=<optimized out>, msg=...) at /cargo/registry/src/index.crates.io-6f17d22bba15001f/crossbeam-channel-0.5.8/src/channel.rs:439
        chan = <optimized out>
#6  0x0000aaaac9e8b14c in mz_kafka_util::client::MzClientContext::record_error (self=<optimized out>, msg=...) at src/kafka-util/src/client.rs:82
        err = mz_kafka_util::client::MzKafkaError::InvalidCredentials
#7  0x0000aaaac9e8beac in mz_kafka_util::client::{impl#3}::log (self=0x0, level=rdkafka::config::RDKafkaLogLevel::Warning, fac=..., log_message=...) at src/kafka-util/src/client.rs:211
No locals.
#8  0x0000aaaac784e248 in rdkafka::client::native_log_cb<mz_kafka_util::client::TunnelingClientContext<mz_storage::source::kafka::GlueConsumerContext>> (client=<optimized out>, level=<optimized out>, fac=<optimized out>, buf=<optimized out>) at /cargo/git/checkouts/rust-rdkafka-545566655b063b24/c729f89/src/client.rs:489
        context = 0xffff8bb34d10
        log_message = alloc::borrow::Cow<str>::Borrowed("[thrd:kafka:9092/bootstrap]: kafka:9092/1: Connection setup timed out in state APIVERSION_QUERY (after 30000ms in state APIVERSION_QUERY)")
        fac = alloc::borrow::Cow<str>::Borrowed("FAIL")
#9  0x0000aaaacb6b9184 in rd_kafka_log_buf (buf=0xffff57418c80 "[thrd:kafka:9092/bootstrap]: kafka:9092/1: Connection setup timed out in state APIVERSION_QUERY (after 30000ms in state APIVERSION_QUERY)", fac=0xaaaac3700bd8 "FAIL", ctx=0, level=4, rk=0xffff712f4c00, conf=0xffff712f4d48) at /cargo/git/checkouts/rust-rdkafka-545566655b063b24/c729f89/rdkafka-sys/librdkafka/src/rdkafka.c:273
No locals.
#10 rd_kafka_log0 (conf=0xffff712f4d48, rk=0xffff712f4c00, extra=extra@entry=0x0, level=level@entry=4, ctx=ctx@entry=0, fac=fac@entry=0xaaaac3700bd8 "FAIL", fmt=fmt@entry=0xaaaac3735a18 "%s: %s") at /cargo/git/checkouts/rust-rdkafka-545566655b063b24/c729f89/rdkafka-sys/librdkafka/src/rdkafka.c:317
        buf = "[thrd:kafka:9092/bootstrap]: kafka:9092/1: Connection setup timed out in state APIVERSION_QUERY (after 30000ms in state APIVERSION_QUERY)\000AW\377\377", '\000' <repeats 14 times>, "\377\377\000\000\000\000\000\000\000\000\000\000\340\377\377\377\000\000\000\000\000\000\000\000\252\252\000\000\000\000\000\000\377\377\000\000J\370s\303\252\252\000\000"...
        ap = {__stack = 0xffff57419510, __gr_top = 0xffff57419510, __vr_top = 0xffff57419500, __gr_offs = -8, __vr_offs = -128}
        elen = <optimized out>
        of = <optimized out>
#11 0x0000aaaacb6c4928 in rd_kafka_broker_set_error (rkb=rkb@entry=0xffff75fb3600, level=level@entry=4, err=err@entry=RD_KAFKA_RESP_ERR__TRANSPORT, fmt=fmt@entry=0xaaaac36fd810 "Connection setup timed out in state %s", ap=<error reading variable: Cannot access memory at address 0x27>) at /cargo/git/checkouts/rust-rdkafka-545566655b063b24/c729f89/rdkafka-sys/librdkafka/src/rdkafka_broker.c:522
        errstr = "Connection setup timed out in state APIVERSION_QUERY (after 30000ms in state APIVERSION_QUERY)\000\000p\230AW\377\377\000\000@\230AW\377\377\000\000\330\377\377\377\200\377\377\377\000U\026I$\264\262Q\000Pmaterialize-mzcompose-us-east-00-0000-00000000 \230AW\377\377\000\000\254\235v\313\252\252\000\000\300\215\273\213\377\377\000\000\000\000\000\000\000\000\000\000"...
        extra = "after 30000ms in state APIVERSION_QUERY\000\000L/q\377\377\000\000H6\373u\377\377\000\000\264\034s\313\252\252\000\000p\230AW\377\377\000\000p\230AW\377\377\000\000@\230AW\377\377\000\000\330\377\377\377\200\377\377\377\000\200S~\377\377\000\000\001\200\255\373\000\000\000\000\240\310AW\377\377\000\000\240\310AW\377\377\000"
        of = <optimized out>
        ofe = <optimized out>
        identical = <optimized out>
        suppress = <optimized out>
        state_duration_ms = <optimized out>
#12 0x0000aaaacb6c56b8 in rd_kafka_broker_fail (rkb=rkb@entry=0xffff75fb3600, level=level@entry=4, err=err@entry=RD_KAFKA_RESP_ERR__TRANSPORT, fmt=fmt@entry=0xaaaac36fd810 "Connection setup timed out in state %s") at /cargo/git/checkouts/rust-rdkafka-545566655b063b24/c729f89/rdkafka-sys/librdkafka/src/rdkafka_broker.c:569
        ap = {__stack = 0xffff57419bc0, __gr_top = 0xffff57419bc0, __vr_top = 0xffff57419ba0, __gr_offs = -32, __vr_offs = -128}
        tmpq_waitresp = {rkbq_bufs = {tqh_first = 0x0, tqh_last = 0xffff00000008}, rkbq_cnt = {val = 5}, rkbq_msg_cnt = {val = 0}}
        tmpq = {rkbq_bufs = {tqh_first = 0xdca6012d, tqh_last = 0xe75}, rkbq_cnt = {val = 866853949}, rkbq_msg_cnt = {val = 0}}
        old_state = <optimized out>
        __FUNCTION__ = "rd_kafka_broker_fail"
#13 0x0000aaaacb6d447c in rd_kafka_broker_thread_main (arg=arg@entry=0xffff75fb3600) at /cargo/git/checkouts/rust-rdkafka-545566655b063b24/c729f89/rdkafka-sys/librdkafka/src/rdkafka_broker.c:5370
        backoff = <optimized out>
        r = <optimized out>
        orig_state = RD_KAFKA_BROKER_STATE_APIVERSION_QUERY
        rkb = 0xffff75fb3600
        rk = 0xffff712f4c00
        redo = <optimized out>
#14 0x0000aaaacb769cec in _thrd_wrapper_function (aArg=<optimized out>) at /cargo/git/checkouts/rust-rdkafka-545566655b063b24/c729f89/rdkafka-sys/librdkafka/src/tinycthread.c:576
        fun = 0xaaaacb6d3be0 <rd_kafka_broker_thread_main>
        arg = 0xffff75fb3600
        res = 0
        ti = <optimized out>
#15 0x0000ffff9c2fd5c8 in __pthread_mutex_lock_full (mutex=0xffff7e9c26de) at ./nptl/pthread_mutex_lock.c:241
        newval = 2126358720
        assume_other_futex_waiters = 1453391872
        oldval = 0
        id = 2124261568
        __PRETTY_FUNCTION__ = "\b\004\000\002\b\004\000\n\000\000\002\002\000\000\002\n\b\000\002\002\b\000\002\n\000\004"
#16 0x0000ffff9c365d1c in __libc_accept (fd=<optimized out>, addr=..., len=<optimized out>) at ../sysdeps/unix/sysv/linux/accept.c:32
No locals.

Full backtrace: https://gist.github.com/def-/859ee9dd8097c16fd1dfb39bb69ee33b

def- avatar Feb 06 '24 12:02 def-

This is still happening: https://buildkite.com/materialize/nightlies/builds/6468#018dad49-70eb-493a-8f90-89326e1a1bfc

def- avatar Feb 15 '24 16:02 def-

I scanned all test runs in CI in the last 2 weeks to see if this is still happening. It actually happened again on Saturday: https://buildkite.com/materialize/nightlies/builds/6598#018ddaa1-ae42-4d54-97a3-72fcac31ea5a

Core was generated by `/usr/local/bin/clusterd --storage-controller-listen-addr=/tmp/614676793cc60fb68'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_mutexattr_getprioceiling (attr=<optimized out>, prioceiling=<optimized out>) at ./nptl/pthread_mutexattr_getprioceiling.c:46
[Current thread is 1 (LWP 563)]
#0  __pthread_mutexattr_getprioceiling (attr=<optimized out>, prioceiling=<optimized out>) at ./nptl/pthread_mutexattr_getprioceiling.c:46
        iattr = <optimized out>
        ceiling = 2111154368
#1  0x0000aaaad77c47bc in core::ptr::drop_in_place<std::sys::unix::thread::Thread> () at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ptr/mod.rs:507
No locals.
#2  core::ptr::drop_in_place<std::thread::JoinInner<()>> () at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ptr/mod.rs:507
No locals.
#3  core::ptr::drop_in_place<std::thread::JoinHandle<()>> () at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ptr/mod.rs:507
No locals.
#4  core::ptr::drop_in_place<mz_ore::thread::UnparkOnDropHandle<()>> () at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ptr/mod.rs:507
No locals.
#5  core::ptr::drop_in_place<mz_storage::source::kafka::KafkaSourceReader> () at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ptr/mod.rs:507
No locals.
#6  0x0000ffff8cfe2e40 in ?? ()
No symbol table info available.

I have to add that we also see other mysterious segfaults which happen in CI, but never cause any problems, so they are probably segfaulting during shutdown or server initialization. But this segfault actually causes the test to fail whenever it happens, so I think it's a legitimate problem:

dependencies.td:349:1: error: error sending request for url (http://materialized:6878/api/catalog/dump): connection closed before message completed: connection closed before message completed

def- avatar Feb 28 '24 07:02 def-

I managed to reproduce this (once so far) on my dev server, which is x86-64, so it's not longer an aarch64 migration blocker!

[10484465.596861] timely:work-4[2804728]: segfault at 7f584d651948 ip 00007f5941186c30 sp 00007f5921be3fb8 error 4 in libc.so.6[7f5941119000+195000] likely on CPU 14 (core 6, socket 0)
[10484465.596877] Code: e8 45 42 fa ff 0f 1f 44 00 00 f3 0f 1e fa 48 85 ff 74 37 8b 87 d0 02 00 00 85 c0 78 2d 31 c0 f0 48 0f b1 bf 28 06 00 00 75 26 <8b> 87 08 03 00 00 83 e0 10 75 05 c3 0f 1f 40 00 48 83 ec 08 e8 f7

The trick was to create more parallelism/load on the machine:

$ bin/mzcompose --find testdrive down && bin/mzcompose --find testdrive run default --default-size=16
[...]
--- avro-resolution-logical-type-default.td
Connecting to PostgreSQL server at postgres://mz_system:materialize@materialized:6877...
> DROP DATABASE materialize
Creating Kafka topic testdrive-resolution-1430454958 with partition count of 1
Ingesting data into Kafka topic testdrive-resolution-1430454958 with start_iteration = 0, repeat = 1
Ingesting data into Kafka topic testdrive-resolution-1430454958 with start_iteration = 0, repeat = 1
> CREATE CONNECTION IF NOT EXISTS csr_conn TO CONFLUENT SCHEMA REGISTRY (
  URL 'http://schema-registry:8081/'
);

^^^ +++
--- avro-resolution-more-columns.td
avro-resolution-logical-type-default.td:27:1: error: preparing query failed: connection closed
     |
  26 |
  27 | > CREATE CONNECTION IF NOT EXISTS csr_conn TO CONFLUENT SCHEMA REGISTRY (
     | ^
^^^ +++
--- avro-resolution-no-publish-reader.td

def- avatar Mar 16 '24 07:03 def-

Another occurrence in CI: https://buildkite.com/materialize/test/builds/84612#019050c7-2b2e-44ac-b591-6e715a3447b8

def- avatar Jun 26 '24 05:06 def-

Another occurrence in CI: https://buildkite.com/materialize/nightly/builds/8396#0190921c-c678-4899-a0de-cf98efa63157

nrainer-materialize avatar Jul 08 '24 14:07 nrainer-materialize