arroyo icon indicating copy to clipboard operation
arroyo copied to clipboard

rust_arroyo: transform_and_produce example panics

Open loewenheim opened this issue 2 years ago • 3 comments

Environment

Current master, i.e. https://github.com/getsentry/snuba/commit/fa3cf8880289c609acafcf9db0b790e354719626.

Steps to Reproduce

 cargo run --example transform_and_produce

Expected Result

The example would run without panicking.

Actual Result

running processor. transforming from test_in to test_out
thread 'main' panicked at rust_arroyo/examples/transform_and_produce.rs:87:21:
called `Result::unwrap()` on an `Err` value: PollError
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

loewenheim avatar Nov 15 '23 11:11 loewenheim

I think this might be because of something basic like either the topic not existing, or the consumer group not having a pre-existing offset.

untitaker avatar Nov 15 '23 11:11 untitaker

The base_processor example also produces errors, although they're not visible unless you add some logging:

2023-11-15T11:48:59.002530Z ERROR rdkafka::client: librdkafka: Global error: UnknownTopicOrPartition (Broker: Unknown topic or partition): Subscribed topic not available: test_static: Broker: Unknown topic or partition    
2023-11-15T11:48:59.002648Z ERROR rust_arroyo::processing: poll error error=Message consumption error: UnknownTopicOrPartition (Broker: Unknown topic or partition)

loewenheim avatar Nov 15 '23 11:11 loewenheim

i guess we should mostly fix this to not throw away the inner error information: https://github.com/getsentry/snuba/blob/c821f741fdd03d41dcdb725e0c95c7d41deea7aa/rust_snuba/rust_arroyo/src/processing/mod.rs#L185-L188

and have perhaps some docs on how to create topics?

untitaker avatar Nov 15 '23 11:11 untitaker