rust-rdkafka
rust-rdkafka copied to clipboard
fede1024
Some methods are not provided as async
For example, offsets_for_times, even with a timeout, does not have na async equivalent, and I have to use block_in_place to work around it
Yep. It’s too bad. Unfortunately there’s nothing I can do (that I know of!) unless upstream exposes some new APIs: https://github.com/edenhill/librdkafka/issues/3212
@benesch I pinged upstream on that ticket. It seems like it might be awhile before this is available, however. I think we could make use of Tokio's blocking executor in the interim. Yes, it's not as efficient as a true async method would be but these methods don't seem to be performance critical. I can see about providing wrappers around offsets_for_times (I've needed this in the past) and the list mentioned in edenhill/librdkafka#3212 .
@benesch I decided to start work on adding async wrappers on the Rust side. I'm taking the approach of 2 new traits: rdkafka::consumer::AsyncConsumer and rdkafka::producer::AsyncProducer to mirror the sync versions. That way if folks want to opt in, they control which implementation they want via bringing the different traits into scope. That means pushing knowledge of the specific AsyncRuntime down from StreamConsumer to BaseConsumer, and adding AsyncRuntime.spawn_blocking to push the blocking calls to a background thread.
@davidblewett Have you made some progress on AsyncConsumer? I'd be happy to give you a hand.
@guilload I started working on this, but haven't been able to get back to it. I would be happy to review if you want to work on it.
Do we still need the Consumer trait once we have migrated the blocking methods to async fn? The purpose of the Consumer trait seems to be aligning interfaces between BaseConsumer and StreamConsumer. Since their interfaces are going to differ, and Rust currently lacks features like keyword generic, is it essential to introduce a new trait like AsyncConsumer solely for StreamConsumer? It's also worth noting that BaseConsumer will be the only implementor of the Consumer trait. Therefore, should we consider deprecating the traits and directly implementing methods on the structs instead?
