Asyncify all the things?

[benesch]

There is a growing suite of librdkafka APIs that block on network requests:

• rd_kafka_metadata (and friends)
• rd_kafka_seek
• rd_kafka_init_transactions, rd_kafka_commit_transaction, et al.

Making rd_kafka_metadata async is tracked by #3085, but I didn't see any issues about async-ifying the seek or producer transactions API, so I wanted to file this.

The implementation of these functions all seem to use rdkafka's internal async APIs, so it feels like we're only an inch or two away from exposing that async nature to the client.

The impetus here is that I'd like to better integrate these APIs into rust-rdkafka. In Rust, the async/await design is very strict: you are never permitted to block in an async task, as it will prevent potentially all other work from running. So to wire these up correctly, we need librdkafka to expose callbacks or some such, rather than blocking.

For what it's worth, the design of the admin API has worked out great. All of those APIs are properly async in rust-rdkafka.

Do you have any plans, @edenhill, to expose async versions of these APIs? Would you be open to contributions that added async versions of those APIs that used the same design as the admin API?

Also, if there already exists some API for this, apologies! I'll be delighted to get that wired up in rust-rdkafka.

[davidblewett]

@edenhill any thoughts on this? While it might be feasible to wrap calls to these APIs in an async context, it won't be the most efficient and could cause complications in the calling Rust code.

[benesch]

Heh, great timing to bring this up again. We're currently evaluating the best way to work around this at Materialize. I'm not hopeful that there's a short path to fixing this here (see https://github.com/MaterializeInc/materialize/issues/6035#issuecomment-1004545078), because I imagine @edenhill and Confluent have other priorities. But I'd love to be proven wrong on that front.

[edenhill]

If there's ever a second incarnation of the public API we'll provide everything as async, with support for cancellation, and then make high-level blocking funtions on top of that too, for the standard use-case.

But this is beyond the current horizon.

[hassila]

That would be awesome also for Swift and async/await there similar to Rust. CC: @FranzBusch