snuba
snuba copied to clipboard
Published
20 hours ago •
getsentry
getsentry
`rust_snuba` should automatically restart or retries the connection after panicked
Environment
self-hosted, v25.3.0
Steps to Reproduce
- Put a container on the
docker-compose.yml - Start it
- Wait for something to go wrong
- ???
- Profit
Docker Compose config:
snuba-eap-spans-consumer:
<<: *snuba_defaults
command: rust-consumer --storage eap_spans --consumer-group snuba-eap-spans-consumers --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset --use-rust-processor
profiles:
- feature-complete
Okay jokes aside. I was investigating why my Sentry instance got the disk maxed out. I moved every storage to S3, therefore it shouldn't behave like that. It turned out, the Kafka volume consumed lots of storage. Then I investigate more on what topic made this issue. It turns out I have a lag on the snuba-spans topic for the snuba-eap-spans-consumer group. See image below:
Then, all I did was to check on the container's log, I found out that the arroyo thread crashed, yet the instance was not automatically restarted.
Expected Result
It should be automatically restarted, since if the consumer thread panicked, it would kill the container, therefore invoking Docker's auto-restart thingy.
Actual Result
Container logs:
snuba-eap-spans-consumer-1 | 2025-04-05 08:54:45,518 Initializing Snuba...
snuba-eap-spans-consumer-1 | 2025-04-05 08:54:51,655 Snuba initialization took 6.138266668654978s
snuba-eap-spans-consumer-1 | 2025-04-05 08:54:52,442 Initializing Snuba...
snuba-eap-spans-consumer-1 | 2025-04-05 08:54:58,575 Snuba initialization took 6.134214653633535s
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-05T08:54:58.588017Z","level":"INFO","fields":{"message":"Storage: eap_spans, ClickHouse Table Name: eap_spans_2_local, Message Processor: MessageProcessorConfig { python_class_name: \"EAPSpansMessageProcessor\", python_module: \"snuba.datasets.processors.spans_v2_processor\" }, ClickHouse host: clickhouse, ClickHouse port: 9000, ClickHouse HTTP port: 8123, ClickHouse database: default"},"target":"rust_snuba::consumer"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-05T08:54:58.588127Z","level":"INFO","fields":{"message":"Starting consumer for \"eap_spans\"","storage":"eap_spans"},"target":"rust_snuba::consumer"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-05T08:54:58.624971Z","level":"INFO","fields":{"message":"Kafka consumer member id: \"rdkafka-692d7d1a-e8bb-4b86-b8b9-0b2676fa3722\""},"target":"sentry_arroyo::backends::kafka"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-05T08:54:58.625262Z","level":"INFO","fields":{"message":"New partitions assigned: {Partition { topic: Topic(\"snuba-spans\"), index: 0 }: 205674898}"},"target":"sentry_arroyo::processing"}
snuba-eap-spans-consumer-1 | thread '<unnamed>' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/backends/kafka/mod.rs:122:52:
snuba-eap-spans-consumer-1 | called `Result::unwrap()` on an `Err` value: KafkaError (Consumer commit error: UnknownMemberId (Broker: Unknown member))
snuba-eap-spans-consumer-1 | stack backtrace:
snuba-eap-spans-consumer-1 | 0: rust_begin_unwind
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
snuba-eap-spans-consumer-1 | 1: core::panicking::panic_fmt
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
snuba-eap-spans-consumer-1 | 2: core::result::unwrap_failed
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/result.rs:1700:5
snuba-eap-spans-consumer-1 | 3: core::result::Result<T,E>::unwrap
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/result.rs:1104:23
snuba-eap-spans-consumer-1 | 4: sentry_arroyo::backends::kafka::commit_impl
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/backends/kafka/mod.rs:122:5
snuba-eap-spans-consumer-1 | 5: <sentry_arroyo::backends::kafka::KafkaConsumer<C> as sentry_arroyo::backends::Consumer<sentry_arroyo::backends::kafka::types::KafkaPayload,C>>::commit_offsets
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/backends/kafka/mod.rs:493:9
snuba-eap-spans-consumer-1 | 6: sentry_arroyo::processing::StreamProcessor<TPayload>::_run_once
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/processing/mod.rs:317:17
snuba-eap-spans-consumer-1 | 7: sentry_arroyo::processing::StreamProcessor<TPayload>::run_once::{{closure}}
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/processing/mod.rs:243:49
snuba-eap-spans-consumer-1 | 8: core::ops::function::FnOnce::call_once
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5
snuba-eap-spans-consumer-1 | 9: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panic/unwind_safe.rs:272:9
snuba-eap-spans-consumer-1 | 10: std::panicking::try::do_call
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40
snuba-eap-spans-consumer-1 | 11: std::panicking::try
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19
snuba-eap-spans-consumer-1 | 12: std::panic::catch_unwind
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14
snuba-eap-spans-consumer-1 | 13: sentry_arroyo::processing::StreamProcessor<TPayload>::run_once
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/processing/mod.rs:243:9
snuba-eap-spans-consumer-1 | 14: sentry_arroyo::processing::StreamProcessor<TPayload>::run
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/processing/mod.rs:438:29
snuba-eap-spans-consumer-1 | 15: rust_snuba::consumer::consumer_impl
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:311:25
snuba-eap-spans-consumer-1 | 16: rust_snuba::consumer::consumer::{{closure}}
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:52:9
snuba-eap-spans-consumer-1 | 17: pyo3::marker::Python::allow_threads
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/marker.rs:473:9
snuba-eap-spans-consumer-1 | 18: rust_snuba::consumer::consumer
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:51:5
snuba-eap-spans-consumer-1 | 19: rust_snuba::consumer::_::__pyfunction_consumer
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:29:1
snuba-eap-spans-consumer-1 | 20: pyo3::impl_::trampoline::fastcall_with_keywords::{{closure}}
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/impl_/trampoline.rs:41:35
snuba-eap-spans-consumer-1 | 21: pyo3::impl_::trampoline::trampoline_inner::{{closure}}
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/impl_/trampoline.rs:204:54
snuba-eap-spans-consumer-1 | 22: std::panicking::try::do_call
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40
snuba-eap-spans-consumer-1 | 23: std::panicking::try
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19
snuba-eap-spans-consumer-1 | 24: std::panic::catch_unwind
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14
snuba-eap-spans-consumer-1 | 25: pyo3::impl_::trampoline::trampoline_inner
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/impl_/trampoline.rs:204:9
snuba-eap-spans-consumer-1 | 26: pyo3::impl_::trampoline::fastcall_with_keywords
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/impl_/trampoline.rs:41:13
snuba-eap-spans-consumer-1 | 27: rust_snuba::consumer::_::<impl rust_snuba::consumer::consumer::MakeDef>::DEF::trampoline
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:29:1
snuba-eap-spans-consumer-1 | 28: <unknown>
snuba-eap-spans-consumer-1 | 29: PyObject_Vectorcall
snuba-eap-spans-consumer-1 | 30: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 31: <unknown>
snuba-eap-spans-consumer-1 | 32: <unknown>
snuba-eap-spans-consumer-1 | 33: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 34: <unknown>
snuba-eap-spans-consumer-1 | 35: <unknown>
snuba-eap-spans-consumer-1 | 36: <unknown>
snuba-eap-spans-consumer-1 | 37: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 38: <unknown>
snuba-eap-spans-consumer-1 | 39: <unknown>
snuba-eap-spans-consumer-1 | 40: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 41: <unknown>
snuba-eap-spans-consumer-1 | 42: _PyObject_FastCallDictTstate
snuba-eap-spans-consumer-1 | 43: _PyObject_Call_Prepend
snuba-eap-spans-consumer-1 | 44: <unknown>
snuba-eap-spans-consumer-1 | 45: _PyObject_MakeTpCall
snuba-eap-spans-consumer-1 | 46: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 47: <unknown>
snuba-eap-spans-consumer-1 | 48: PyEval_EvalCode
snuba-eap-spans-consumer-1 | 49: <unknown>
snuba-eap-spans-consumer-1 | 50: <unknown>
snuba-eap-spans-consumer-1 | 51: <unknown>
snuba-eap-spans-consumer-1 | 52: _PyRun_SimpleFileObject
snuba-eap-spans-consumer-1 | 53: _PyRun_AnyFileObject
snuba-eap-spans-consumer-1 | 54: Py_RunMain
snuba-eap-spans-consumer-1 | 55: Py_BytesMain
snuba-eap-spans-consumer-1 | 56: <unknown>
snuba-eap-spans-consumer-1 | 57: __libc_start_main
snuba-eap-spans-consumer-1 | 58: _start
snuba-eap-spans-consumer-1 | note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.870420Z","level":"INFO","fields":{"message":"Caught error, terminating strategy"},"target":"sentry_arroyo::processing"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.873798Z","level":"WARN","fields":{"message":"librdkafka: SESSTMOUT [thrd:main]: Consumer group session timed out (in join-state steady) after 30002 ms without a successful response from the group coordinator (broker 0, last error was Success): revoking assignment and rejoining group"},"target":"sentry_arroyo::backends::kafka"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.874669Z","level":"WARN","fields":{"message":"librdkafka: REQTMOUT [thrd:GroupCoordinator]: GroupCoordinator/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests"},"target":"sentry_arroyo::backends::kafka"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.874683Z","level":"ERROR","fields":{"message":"librdkafka: FAIL [thrd:GroupCoordinator]: GroupCoordinator: kafka:9092: 1 request(s) timed out: disconnect (after 189229996ms in state UP)"},"target":"sentry_arroyo::backends::kafka"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.874690Z","level":"WARN","fields":{"message":"librdkafka: MAXPOLL [thrd:main]: Application maximum poll interval (30000ms) exceeded by 9271ms (adjust max.poll.interval.ms for long-running message processing): leaving group"},"target":"sentry_arroyo::backends::kafka"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.874700Z","level":"ERROR","fields":{"message":"librdkafka: Global error: OperationTimedOut (Local: Timed out): GroupCoordinator: kafka:9092: 1 request(s) timed out: disconnect (after 189229996ms in state UP)","error":"Global error: OperationTimedOut (Local: Timed out)"},"target":"sentry_arroyo::backends::kafka"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.875061Z","level":"INFO","fields":{"message":"Partitions to revoke: [Partition { topic: Topic(\"snuba-spans\"), index: 0 }]"},"target":"sentry_arroyo::processing"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.875082Z","level":"INFO","fields":{"message":"Joining strategy"},"target":"sentry_arroyo::processing"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.875470Z","level":"WARN","fields":{"message":"Timeout Some(0ns) reached while waiting for tasks to finish"},"target":"sentry_arroyo::processing::strategies::reduce"}
snuba-eap-spans-consumer-1 | {"timestamp":"2025-04-07T13:28:50.875504Z","level":"INFO","fields":{"message":"Committing offsets"},"target":"sentry_arroyo::processing"}
snuba-eap-spans-consumer-1 | thread '<unnamed>' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/backends/kafka/mod.rs:122:52:
snuba-eap-spans-consumer-1 | called `Result::unwrap()` on an `Err` value: KafkaError (Consumer commit error: UnknownMemberId (Broker: Unknown member))
snuba-eap-spans-consumer-1 | stack backtrace:
snuba-eap-spans-consumer-1 | 0: rust_begin_unwind
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
snuba-eap-spans-consumer-1 | 1: core::panicking::panic_fmt
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
snuba-eap-spans-consumer-1 | 2: core::result::unwrap_failed
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/result.rs:1700:5
snuba-eap-spans-consumer-1 | 3: core::result::Result<T,E>::unwrap
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/result.rs:1104:23
snuba-eap-spans-consumer-1 | 4: sentry_arroyo::backends::kafka::commit_impl
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/backends/kafka/mod.rs:122:5
snuba-eap-spans-consumer-1 | 5: <sentry_arroyo::backends::kafka::OffsetCommitter<C> as sentry_arroyo::backends::CommitOffsets>::commit
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/backends/kafka/mod.rs:132:9
snuba-eap-spans-consumer-1 | 6: <sentry_arroyo::processing::Callbacks<TPayload> as sentry_arroyo::backends::AssignmentCallbacks>::on_revoke
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/processing/mod.rs:161:35
snuba-eap-spans-consumer-1 | 7: <sentry_arroyo::backends::kafka::CustomContext<C> as rdkafka::consumer::ConsumerContext>::rebalance
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/backends/kafka/mod.rs:226:17
snuba-eap-spans-consumer-1 | 8: rdkafka::consumer::base_consumer::BaseConsumer<C>::handle_rebalance_event
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rdkafka-0.37.0/src/consumer/base_consumer.rs:208:17
snuba-eap-spans-consumer-1 | 9: rdkafka::consumer::base_consumer::BaseConsumer<C>::poll_queue
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rdkafka-0.37.0/src/consumer/base_consumer.rs:150:29
snuba-eap-spans-consumer-1 | 10: rdkafka::consumer::base_consumer::BaseConsumer<C>::poll
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rdkafka-0.37.0/src/consumer/base_consumer.rs:120:9
snuba-eap-spans-consumer-1 | 11: <rdkafka::consumer::base_consumer::BaseConsumer<C> as core::ops::drop::Drop>::drop
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rdkafka-0.37.0/src/consumer/base_consumer.rs:775:26
snuba-eap-spans-consumer-1 | 12: core::ptr::drop_in_place<rdkafka::consumer::base_consumer::BaseConsumer<sentry_arroyo::backends::kafka::CustomContext<sentry_arroyo::processing::Callbacks<sentry_arroyo::backends::kafka::types::KafkaPayload>>>>
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ptr/mod.rs:574:1
snuba-eap-spans-consumer-1 | 13: core::ptr::drop_in_place<sentry_arroyo::backends::kafka::KafkaConsumer<sentry_arroyo::processing::Callbacks<sentry_arroyo::backends::kafka::types::KafkaPayload>>>
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ptr/mod.rs:574:1
snuba-eap-spans-consumer-1 | 14: core::ptr::drop_in_place<alloc::boxed::Box<dyn sentry_arroyo::backends::Consumer<sentry_arroyo::backends::kafka::types::KafkaPayload,sentry_arroyo::processing::Callbacks<sentry_arroyo::backends::kafka::types::KafkaPayload>>>>
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ptr/mod.rs:574:1
snuba-eap-spans-consumer-1 | 15: core::ptr::drop_in_place<sentry_arroyo::processing::StreamProcessor<sentry_arroyo::backends::kafka::types::KafkaPayload>>
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ptr/mod.rs:574:1
snuba-eap-spans-consumer-1 | 16: sentry_arroyo::processing::StreamProcessor<TPayload>::run
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sentry_arroyo-2.19.11/rust-arroyo/src/processing/mod.rs:452:5
snuba-eap-spans-consumer-1 | 17: rust_snuba::consumer::consumer_impl
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:311:25
snuba-eap-spans-consumer-1 | 18: rust_snuba::consumer::consumer::{{closure}}
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:52:9
snuba-eap-spans-consumer-1 | 19: pyo3::marker::Python::allow_threads
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/marker.rs:473:9
snuba-eap-spans-consumer-1 | 20: rust_snuba::consumer::consumer
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:51:5
snuba-eap-spans-consumer-1 | 21: rust_snuba::consumer::_::__pyfunction_consumer
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:29:1
snuba-eap-spans-consumer-1 | 22: pyo3::impl_::trampoline::fastcall_with_keywords::{{closure}}
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/impl_/trampoline.rs:41:35
snuba-eap-spans-consumer-1 | 23: pyo3::impl_::trampoline::trampoline_inner::{{closure}}
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/impl_/trampoline.rs:204:54
snuba-eap-spans-consumer-1 | 24: std::panicking::try::do_call
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40
snuba-eap-spans-consumer-1 | 25: std::panicking::try
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19
snuba-eap-spans-consumer-1 | 26: std::panic::catch_unwind
snuba-eap-spans-consumer-1 | at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14
snuba-eap-spans-consumer-1 | 27: pyo3::impl_::trampoline::trampoline_inner
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/impl_/trampoline.rs:204:9
snuba-eap-spans-consumer-1 | 28: pyo3::impl_::trampoline::fastcall_with_keywords
snuba-eap-spans-consumer-1 | at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/impl_/trampoline.rs:41:13
snuba-eap-spans-consumer-1 | 29: rust_snuba::consumer::_::<impl rust_snuba::consumer::consumer::MakeDef>::DEF::trampoline
snuba-eap-spans-consumer-1 | at ./rust_snuba/src/consumer.rs:29:1
snuba-eap-spans-consumer-1 | 30: <unknown>
snuba-eap-spans-consumer-1 | 31: PyObject_Vectorcall
snuba-eap-spans-consumer-1 | 32: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 33: <unknown>
snuba-eap-spans-consumer-1 | 34: <unknown>
snuba-eap-spans-consumer-1 | 35: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 36: <unknown>
snuba-eap-spans-consumer-1 | 37: <unknown>
snuba-eap-spans-consumer-1 | 38: <unknown>
snuba-eap-spans-consumer-1 | 39: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 40: <unknown>
snuba-eap-spans-consumer-1 | 41: <unknown>
snuba-eap-spans-consumer-1 | 42: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 43: <unknown>
snuba-eap-spans-consumer-1 | 44: _PyObject_FastCallDictTstate
snuba-eap-spans-consumer-1 | 45: _PyObject_Call_Prepend
snuba-eap-spans-consumer-1 | 46: <unknown>
snuba-eap-spans-consumer-1 | 47: _PyObject_MakeTpCall
snuba-eap-spans-consumer-1 | 48: _PyEval_EvalFrameDefault
snuba-eap-spans-consumer-1 | 49: <unknown>
snuba-eap-spans-consumer-1 | 50: PyEval_EvalCode
snuba-eap-spans-consumer-1 | 51: <unknown>
snuba-eap-spans-consumer-1 | 52: <unknown>
snuba-eap-spans-consumer-1 | 53: <unknown>
snuba-eap-spans-consumer-1 | 54: _PyRun_SimpleFileObject
snuba-eap-spans-consumer-1 | 55: _PyRun_AnyFileObject
snuba-eap-spans-consumer-1 | 56: Py_RunMain
snuba-eap-spans-consumer-1 | 57: Py_BytesMain
snuba-eap-spans-consumer-1 | 58: <unknown>
snuba-eap-spans-consumer-1 | 59: __libc_start_main
snuba-eap-spans-consumer-1 | 60: _start
snuba-eap-spans-consumer-1 | note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
