kafka-delta-ingest icon indicating copy to clipboard operation
kafka-delta-ingest copied to clipboard

Published 20 hours ago • verified publisher icon delta-io

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value

Open huyhandes opened this issue 2 years ago • 3 comments

I tried to reproduce the example with amazon MSK and my S3 but failed. Debug log:

2023-08-30T04:42:37 [INFO] - tester-mode-on: Ingesting messages from deltalake-test-ingest Kafka topic to s3a://datalake-test/deltalake-test-ingest Delta table
2023-08-30T04:42:37 [INFO] - tester-mode-on: Using options: [allowed_latency=60,max_messages_per_batch=5000,min_bytes_per_file=134217728,write_checkpoints=false]
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BROKER [thrd:app]: GroupCoordinator: Added new broker with NodeId -1
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKMAIN [thrd:GroupCoordinator]: GroupCoordinator: Enter main broker thread
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BROKER [thrd:app]: b-1.OUR-KAFKA-HOST:9092/bootstrap: Added new broker with NodeId -1
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKMAIN [thrd::0/internal]: :0/internal: Enter main broker thread
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BROKER [thrd:app]: b-2.OUR-KAFKA-HOST:9092/bootstrap: Added new broker with NodeId -1
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKMAIN [thrd:b-1.OUR-KAFKA-HOST:9092/b]: b-1.OUR-KAFKA-HOST:9092/bootstrap: Enter main broker thread
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BROKER [thrd:app]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Added new broker with NodeId -1
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKMAIN [thrd:b-2.OUR-KAFKA-HOST:9092/b]: b-2.OUR-KAFKA-HOST:9092/bootstrap: Enter main broker thread
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BROKER [thrd:app]: b-4.OUR-KAFKA-HOST:9092/bootstrap: Added new broker with NodeId -1
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: INIT [thrd:app]: librdkafka v2.2.0 (0x20200ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, GCC GXX PKGCONFIG INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x282)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKMAIN [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Enter main broker thread
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKMAIN [thrd:b-4.OUR-KAFKA-HOST:9092/b]: b-4.OUR-KAFKA-HOST:9092/bootstrap: Enter main broker thread
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: CONNECT [thrd:main]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Selected for cluster connection: coordinator query (broker has 0 connection attempt(s))
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: CONNECT [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Received CONNECT op
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: STATE [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: CONNECT [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: broker in state TRY_CONNECT connecting
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: STATE [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
2023-08-30T04:42:37 [INFO] - tester-mode-on: Using Static credential provider
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: add_pem_file processed 129 valid and 0 invalid certs
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: loading checkpoint from _delta_log/_last_checkpoint
thread 'main' panicked at 'not stream', /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/object_store-0.5.6/src/aws/credential.rs:189:14
stack backtrace:
   0:     0x5599fef1bc84 - std::backtrace_rs::backtrace::libunwind::trace::he648b5c8dd376705
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x5599fef1bc84 - std::backtrace_rs::backtrace::trace_unsynchronized::h5da3e203eef39e9f
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5599fef1bc84 - std::sys_common::backtrace::_print_fmt::h8d28d3f20588ae4c
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x5599fef1bc84 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd9a5b0c9c6b058c0
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x5599fe54e1bf - core::fmt::rt::Argument::fmt::h0afc04119f252b53
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/fmt/rt.rs:138:9
   5:     0x5599fe54e1bf - core::fmt::write::h50b1b3e73851a6fe
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/fmt/mod.rs:1094:21
   6:     0x5599feee2836 - std::io::Write::write_fmt::h184eaf275e4484f0
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/io/mod.rs:1714:15
   7:     0x5599fef1d0ff - std::sys_common::backtrace::_print::hf58c3a5a25090e71
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x5599fef1d0ff - std::sys_common::backtrace::print::hb9cf0a7c7f077819
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x5599fef1ccc3 - std::panicking::default_hook::{{closure}}::h066adb2e3f3e2c07
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:269:22
  10:     0x5599fef1dcc5 - std::panicking::default_hook::h277fa2776900ff14
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:288:9
  11:     0x5599fef1dcc5 - std::panicking::rust_panic_with_hook::hceaf38da6d9db792
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:705:13
  12:     0x5599fef1d7c2 - std::panicking::begin_panic_handler::{{closure}}::h2bce3ed2516af7df
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:597:13
  13:     0x5599fef1d726 - std::sys_common::backtrace::__rust_end_short_backtrace::h090f3faf8f98a395
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:151:18
  14:     0x5599fef1d711 - rust_begin_unwind
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:593:5
  15:     0x5599fe01fc12 - core::panicking::panic_fmt::h4ec8274704d163a3
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:67:14
  16:     0x5599fe020152 - core::panicking::panic_display::h5ef861b25744765d
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:150:5
  17:     0x5599fe020152 - core::panicking::panic_str::hb2e5b3cf68d2f306
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:134:5
  18:     0x5599fe020152 - core::option::expect_failed::h3de37afca26e8e59
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/option.rs:1952:5
  19:     0x5599fe92fd8e - <reqwest::async_impl::request::RequestBuilder as object_store::aws::credential::CredentialExt>::with_aws_sigv4::h0f57dc36ae29eb36
  20:     0x5599fe8b194b - object_store::aws::client::S3Client::get_request::{{closure}}::h40d8fd9b78abf607
  21:     0x5599fe8b1186 - <object_store::aws::AmazonS3 as object_store::ObjectStore>::get::{{closure}}::ha213cbe47e3cc5b7
  22:     0x5599fe636042 - <deltalake::storage::s3::S3StorageBackend as object_store::ObjectStore>::get::{{closure}}::hf8bae65134071c6f
  23:     0x5599fe5e7b18 - <object_store::prefix::PrefixStore<T> as object_store::ObjectStore>::get::{{closure}}::hb27bf0a15bae6b07
  24:     0x5599fe689243 - <deltalake::storage::DeltaObjectStore as object_store::ObjectStore>::get::{{closure}}::ha27fa50af36203fa
  25:     0x5599fe0a0a45 - deltalake::delta::DeltaTable::get_last_checkpoint::{{closure}}::h42daa366a2a74ecf
  26:     0x5599fe0a2bbd - deltalake::delta::DeltaTable::update::{{closure}}::h5df80b882c67bbe8
  27:     0x5599fe0a2a38 - deltalake::delta::DeltaTable::load::{{closure}}::hd775a8a5aecc216f
  28:     0x5599fe087090 - kafka_delta_ingest::delta_helpers::load_table::{{closure}}::hc2b0a1876bdedcea
  29:     0x5599fe06d570 - kafka_delta_ingest::start_ingest::{{closure}}::h48d97c594f25df7c
  30:     0x5599fe098a3c - tokio::runtime::task::core::Core<T,S>::poll::h97ea35cfb7167f90
  31:     0x5599fe815292 - tokio::runtime::task::raw::poll::hdfd28e12a40e9032
  32:     0x5599fe81cb84 - tokio::runtime::context::runtime::enter_runtime::h10ab6c4e1562baec
  33:     0x5599fe8414e7 - kafka_delta_ingest::main::h43894231b37e2322
  34:     0x5599fe8133e3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hc2b14238c4bc1fc1
  35:     0x5599fe8409db - main
  36:     0x7f8fb86c4d0a - __libc_start_main
  37:     0x5599fe068caa - _start
  38:                0x0 - <unknown>
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: Async metrics receive loop terminated: receiving on an empty and disconnected channel
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: DESTROY [thrd:app]: Terminating instance (destroy flags none (0x0))
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: DESTROY [thrd:main]: Destroy internal
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: DESTROY [thrd:main]: Removing all topics
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: DESTROY [thrd:main]: Sending TERMINATE to b-4.OUR-KAFKA-HOST:9092/bootstrap
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: DESTROY [thrd:main]: Sending TERMINATE to b-3.OUR-KAFKA-HOST:9092/bootstrap
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: DESTROY [thrd:main]: Sending TERMINATE to b-2.OUR-KAFKA-HOST:9092/bootstrap
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: DESTROY [thrd:main]: Sending TERMINATE to b-1.OUR-KAFKA-HOST:9092/bootstrap
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERM [thrd:b-4.OUR-KAFKA-HOST:9092/b]: b-4.OUR-KAFKA-HOST:9092/bootstrap: Received TERMINATE op in state INIT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: DESTROY [thrd:main]: Sending TERMINATE to GroupCoordinator
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERM [thrd:b-2.OUR-KAFKA-HOST:9092/b]: b-2.OUR-KAFKA-HOST:9092/bootstrap: Received TERMINATE op in state INIT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:b-4.OUR-KAFKA-HOST:9092/b]: b-4.OUR-KAFKA-HOST:9092/bootstrap: Client is terminating (after 9ms in state INIT) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERM [thrd:b-1.OUR-KAFKA-HOST:9092/b]: b-1.OUR-KAFKA-HOST:9092/bootstrap: Received TERMINATE op in state INIT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:b-2.OUR-KAFKA-HOST:9092/b]: b-2.OUR-KAFKA-HOST:9092/bootstrap: Client is terminating (after 9ms in state INIT) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERM [thrd:GroupCoordinator]: GroupCoordinator: Received TERMINATE op in state INIT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERM [thrd::0/internal]: :0/internal: Received TERMINATE op in state INIT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:b-1.OUR-KAFKA-HOST:9092/b]: b-1.OUR-KAFKA-HOST:9092/bootstrap: Client is terminating (after 9ms in state INIT) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:GroupCoordinator]: GroupCoordinator: Client is terminating (after 9ms in state INIT) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd::0/internal]: :0/internal: Client is terminating (after 9ms in state INIT) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: STATE [thrd:GroupCoordinator]: GroupCoordinator: Broker changed state INIT -> DOWN
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: STATE [thrd:b-4.OUR-KAFKA-HOST:9092/b]: b-4.OUR-KAFKA-HOST:9092/bootstrap: Broker changed state INIT -> DOWN
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKTERM [thrd:GroupCoordinator]: GroupCoordinator: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: STATE [thrd:b-2.OUR-KAFKA-HOST:9092/b]: b-2.OUR-KAFKA-HOST:9092/bootstrap: Broker changed state INIT -> DOWN
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERMINATE [thrd:GroupCoordinator]: GroupCoordinator: Handle is terminating in state DOWN: 1 refcnts (0x559a00e875e0), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKTERM [thrd:b-2.OUR-KAFKA-HOST:9092/b]: b-2.OUR-KAFKA-HOST:9092/bootstrap: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:GroupCoordinator]: GroupCoordinator: Broker handle is terminating (after 0ms in state DOWN) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: STATE [thrd:b-1.OUR-KAFKA-HOST:9092/b]: b-1.OUR-KAFKA-HOST:9092/bootstrap: Broker changed state INIT -> DOWN
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: STATE [thrd::0/internal]: :0/internal: Broker changed state INIT -> DOWN
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKTERM [thrd:b-4.OUR-KAFKA-HOST:9092/b]: b-4.OUR-KAFKA-HOST:9092/bootstrap: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERMINATE [thrd:b-2.OUR-KAFKA-HOST:9092/b]: b-2.OUR-KAFKA-HOST:9092/bootstrap: Handle is terminating in state DOWN: 1 refcnts (0x559a00e941c0), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKTERM [thrd:b-1.OUR-KAFKA-HOST:9092/b]: b-1.OUR-KAFKA-HOST:9092/bootstrap: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKTERM [thrd::0/internal]: :0/internal: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERMINATE [thrd:b-1.OUR-KAFKA-HOST:9092/b]: b-1.OUR-KAFKA-HOST:9092/bootstrap: Handle is terminating in state DOWN: 1 refcnts (0x559a00e93350), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERMINATE [thrd:b-4.OUR-KAFKA-HOST:9092/b]: b-4.OUR-KAFKA-HOST:9092/bootstrap: Handle is terminating in state DOWN: 1 refcnts (0x559a00e95ff0), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:b-1.OUR-KAFKA-HOST:9092/b]: b-1.OUR-KAFKA-HOST:9092/bootstrap: Broker handle is terminating (after 0ms in state DOWN) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:b-2.OUR-KAFKA-HOST:9092/b]: b-2.OUR-KAFKA-HOST:9092/bootstrap: Broker handle is terminating (after 0ms in state DOWN) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERMINATE [thrd::0/internal]: :0/internal: Handle is terminating in state DOWN: 1 refcnts (0x559a00e926d0), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:b-4.OUR-KAFKA-HOST:9092/b]: b-4.OUR-KAFKA-HOST:9092/bootstrap: Broker handle is terminating (after 0ms in state DOWN) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd::0/internal]: :0/internal: Broker handle is terminating (after 0ms in state DOWN) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: CONNECT [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Connecting to ipv4#10.10.60.135:9092 (plaintext) with socket 9
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERMINATE [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Handle is terminating in state CONNECT: 2 refcnts (0x559a00e950b0), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERM [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Received TERMINATE op in state CONNECT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Client is terminating (after 51ms in state CONNECT) (_DESTROY)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: STATE [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Broker changed state CONNECT -> DOWN
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: BRKTERM [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: TERMINATE [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Handle is terminating in state DOWN: 1 refcnts (0x559a00e950b0), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
2023-08-30T04:42:37 [DEBUG] - tester-mode-on: librdkafka: FAIL [thrd:b-3.OUR-KAFKA-HOST:9092/b]: b-3.OUR-KAFKA-HOST:9092/bootstrap: Broker handle is terminating (after 0ms in state DOWN) (_DESTROY)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(1), ...)', src/main.rs:173:14
stack backtrace:
   0:     0x5599fef1bc84 - std::backtrace_rs::backtrace::libunwind::trace::he648b5c8dd376705
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x5599fef1bc84 - std::backtrace_rs::backtrace::trace_unsynchronized::h5da3e203eef39e9f
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x5599fef1bc84 - std::sys_common::backtrace::_print_fmt::h8d28d3f20588ae4c
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x5599fef1bc84 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd9a5b0c9c6b058c0
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x5599fe54e1bf - core::fmt::rt::Argument::fmt::h0afc04119f252b53
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/fmt/rt.rs:138:9
   5:     0x5599fe54e1bf - core::fmt::write::h50b1b3e73851a6fe
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/fmt/mod.rs:1094:21
   6:     0x5599feee2836 - std::io::Write::write_fmt::h184eaf275e4484f0
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/io/mod.rs:1714:15
   7:     0x5599fef1d0ff - std::sys_common::backtrace::_print::hf58c3a5a25090e71
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x5599fef1d0ff - std::sys_common::backtrace::print::hb9cf0a7c7f077819
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x5599fef1ccc3 - std::panicking::default_hook::{{closure}}::h066adb2e3f3e2c07
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:269:22
  10:     0x5599fef1dcc5 - std::panicking::default_hook::h277fa2776900ff14
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:288:9
  11:     0x5599fef1dcc5 - std::panicking::rust_panic_with_hook::hceaf38da6d9db792
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:705:13
  12:     0x5599fef1d7c2 - std::panicking::begin_panic_handler::{{closure}}::h2bce3ed2516af7df
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:597:13
  13:     0x5599fef1d726 - std::sys_common::backtrace::__rust_end_short_backtrace::h090f3faf8f98a395
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:151:18
  14:     0x5599fef1d711 - rust_begin_unwind
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:593:5
  15:     0x5599fe01fc12 - core::panicking::panic_fmt::h4ec8274704d163a3
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:67:14
  16:     0x5599fe0200e2 - core::result::unwrap_failed::h170bc2721a6c6ff2
                               at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/result.rs:1651:5
  17:     0x5599fe81e39c - tokio::runtime::context::runtime::enter_runtime::h10ab6c4e1562baec
  18:     0x5599fe8414e7 - kafka_delta_ingest::main::h43894231b37e2322
  19:     0x5599fe8133e3 - std::sys_common::backtrace::__rust_begin_short_backtrace::hc2b14238c4bc1fc1
  20:     0x5599fe8409db - main
  21:     0x7f8fb86c4d0a - __libc_start_main
  22:     0x5599fe068caa - _start
  23:                0x0 - <unknown>

huyhandes avatar Aug 30 '23 05:08 huyhandes

This looks like something caused kafka-delta-ingest to fail to configure credentials for interacting with S3. I'm wondering if you could share a bit more about the runtime environment (EC2, Fargate, etc?) and what form of credentials are being configured for kafka-delta-ingest

rtyler avatar Aug 31 '23 15:08 rtyler

This looks like something caused kafka-delta-ingest to fail to configure credentials for interacting with S3. I'm wondering if you could share a bit more about the runtime environment (EC2, Fargate, etc?) and what form of credentials are being configured for kafka-delta-ingest

This is how I ran the example:

export AWS_ENDPOINT_URL=https://s3.ap-southeast-1.amazonaws.com
export AWS_REGION=ap-southeast-1
export AWS_ACCESS_KEY_ID=OUR_KEY
export AWS_SECRET_ACCESS_KEY=OUR_ACCESS_SECRET

docker run -it --rm --network=host \
  -e RUST_LOG="debug" \
  -e AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID} \
  -e AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY} \
  -e AWS_ENDPOINT_URL={AWS_ENDPOINT_URL} \
  -e AWS_REGION={AWS_REGION} \
  -e RUST_BACKTRACE=full \
  kdi:0.1 \
  ingest deltalake-test-ingest s3a://BUCKET/deltalake-test-ingest \
  --allowed_latency 60 \
  --kafka "OUR_KAFKA_1:9093,OUR_KAFKA_2:9093" \
  -K socket.keepalive.enable=true \
  -K metadata.max.age.ms=180000 \
  -K heartbeat.interval.ms=3000 \
  -K session.timeout.ms=30000 \
  -K debug=broker,security,protocol \
  --app_id tester-mode-on \
  --transform "meta.kafka.topic: kafka.topic" \
  --auto_offset_reset earliest

huyhandes avatar Sep 05 '23 02:09 huyhandes

@huyhandes would you be able to try this again with the latest main? We have some refactorings and crate upgrades that have landed this past weekend which might just resolve this issue :thinking:

rtyler avatar Dec 18 '23 16:12 rtyler