streamnative
Cooperation on Pulsar client for Rust
Hi @wyyerd and Stearns,
Hope you’re having a great holiday.
My name is Yu Liu. I'm a Pulsar committer and technical writer.
When browsing the Pulsar website, I found that you have developed a wonderful Rust client for Pulsar and draw much attention. Thanks for this great work!
Since good projects are collaborations between motivated individuals, I'm wondering if you'd like to donate the Rust client to Apache Pulsar, which is beneficial for attracting more programmers to share quality code, appealing to more technical writers to develop docs (I found you've written some guides and I'd like to contribute together), and allowing more users to use it conveniently.
Any thoughts on this? Looking forward to hearing from you.
Merry Christmas with lots of love.
Yu
Hello @Anonymitaet, it would make sense to integrate the Rust client in the Pulsar project and have it used as the official client. So I'm ok with that! How about you @stearnsc ?
What would be the next steps for integration under the Pulsar project?
Yeah, I'm also in support of this. Can you give us a brief rundown of how this would work @Anonymitaet?
Thank you @Geal and @stearnsc.
The first step is to write a PIP for this. You can check PIP 53: Contribute DotPulsar to Apache Pulsar
as an example and send it or anything you want to know (for example, where the code lives, who maintains it, who publishes it, and so on) to [email protected]. PMC will handle and help with the next procedures.
It's a super busy time for us at work right now, but I should get to this in the next few days, most likely.
Hi @stearnsc,
Since you are working on PIP, maybe we can start working on the user guide as well? I'd like to contribute together and I'm wondering which way do you prefer:
#1: contribute the Rust user guide to the Pulsar website yourself (you can take Pulsar client docs and Pulsar client feature matrix as references).
or
#2: I draft one based on Rust readme, request your review and technical inputs, and then submit doc PR to Pulsar repo.
Let me know your thoughts. Thank you.
Hi @Geal @stearnsc
Thanks for your existing docs, which is beneficial for Rust users to get a quick start.
I was intended to draft one but it seems that the existing docs are not enough to create a complete and accurate guide, it would be much helpful if you could help on this, could you please provide more information? Many thanks!
Here are some tips for you to provide more tech inputs, you can take them as references:
- I've created a Pulsar Client Doc Template, which includes writing tips and Pulsar Client Feature Matrix.
- Pulsar Client Feature Matrix is quite helpful for providing complete docs. We are collecting features supported by different clients and will make the contents public once we finish, so could you please help tick items for the Rust client?
- Existing Pulsar Client Docs are good references as well.
As we know, quality docs help users use the project and be able to improve on the project in return. Hope we can work together to build a robust, mature, and successful project. Looking forward to hearing from you soon.
Hi @Geal @stearnsc, any feedback for the client docs and the Pulsar Client Feature Matrix? Thank you
@Anonymitaet I'm finishing up on the 2.0 release then getting on that. It will be a good occasion to review the docs
@Geal Thanks. I'd love to help if needed. Looking forward to the 2.0 release.
@Geal I see Rust 2.0 is released, congratulations! Can we start working on the docs?
@Anonymitaet I'm filling the features matrix, and I have a few questions:
- what am I supposed to add in the "code" and "doc" columns? Links to the relevant parts of the code and docs?
- we don't have a reader yet, but from what I've seen, it's a wrapper around a consumer with non durable subscription and automatic acks? It should be easy enough to add
- what is an interceptor? I see no mention of it in the documentation
- what is a key based batcher?
- could you provide a list of the features that should be added in priority? I have to admit that so far, we developed that client by adding features as we needed them, but a proper roadmap would be better
- for the schemas, they are not integrated in the client, but we provide traits that users can implement to handle it. This will be tricky to integrate because the various serialization libraries do not use the same approaches
@Geal thanks for your work!
what am I supposed to add in the "code" and "doc" columns? Links to the relevant parts of the code and docs?
You can tick off an item ✅ if the code or doc is done. Adding links to the relevant parts of the code and docs is absolutely good.
@codelipenghui could you please help answer the rest technical questions? Many thanks!
@Geal FYI,
we don't have a reader yet, but from what I've seen, it's a wrapper around a consumer with non durable subscription and automatic acks? It should be easy enough to add
Yes.
what is an interceptor? I see no mention of it in the documentation
See ConsumerInterceptor and ProducerInterceptor for reference.
what is a key based batcher?
See BatcherBuilder for reference.
BTW, I found many public methods/fields have no docs. For example,
It's better to add some docs.
could you provide a list of the features that should be added in priority? I have to admit that so far, we developed that client by adding features as we needed them, but a proper roadmap would be better
I have marked some features as high priority here According to my understanding of some of the features which of the high frequency of use
for the schemas, they are not integrated in the client, but we provide traits that users can implement to handle it. This will be tricky to integrate because the various serialization libraries do not use the same approaches
Pulsar supports the built-in schema registry which is very convenient for users, no need to deploy a separate schema registry service. The Java Client, CPP Client, Go Client already support the schema feature. Users also expect to be able to use it in rust client.
alright, I'm opening some issues to track the new features to add.
For compacted topics, does it just need to set readCompacted to true when creating the consumer, or are there some special cases when receiving messages afterwards?
Do you have any advice on how to implement the schema and interacting with the schema registry? This will require communication with the HTTP API right? I'd expect this is the most complex feature to add right now.
Are all of those blockers for the integration into Apache Pulsar, or can the work on them be done in parallel?
For compacted topics, does it just need to set readCompacted to true when creating the consumer, or are there some special cases when receiving messages afterwards?
I think no, just need to set readCompacted to true.
Do you have any advice on how to implement the schema and interacting with the schema registry? This will require communication with the HTTP API right? I'd expect this is the most complex feature to add right now.
No, schema is defined in the protobuf, when creating a producer or consumer, the client can pass the schema data to server.
Are all of those blockers for the integration into Apache Pulsar, or can the work on them be done in parallel?
No, it's not a blocker, we can work on them in parallel. We need to provide documentation for the rust client so that users can find the tutorial on the Pulsar website. I think all of them are not blockers.
alright then. I'm adding more content to the docs, but first I'll go through the source and add more examples, as they'll appear directly on https://docs.rs/pulsar/. Usually Rust developers will go straight to docs.rs, so we'll have the source of truth there, and copy examples over to the pulsar website
Ok, @Anonymitaet I think we can add a link in the pulsar website point to https://docs.rs/pulsar? so that we don't need to maintain multiple copies
there's more doc comments and examples now, but I'm holding off on a release until I've fixed a bug where some consumers do not get messages
@Geal yes I find more contents filled here. You can ping me once you need a doc review.
Moreover, Pulsar community is updating the Pulsar Feature Matrix for Clients, you can take it as a reference.
@Anonymitaet I just published version 2.0.1 with a fix that we were testing, and more documentation. It should be available soon on https://docs.rs/pulsar/2.0.1/pulsar/
A colleague at Clever Cloud is advancing on a Reader implementation
Hi @Geal thanks for the updates.
I've checked Pulsar Client Feature Matrix - Rust, it seems that the docs on https://docs.rs/pulsar/2.0.1/pulsar/ are not synced with the codes?
To allow users to be kept with the latest info clearly, it would be much helpful to create complete docs based on the doc architecture in Pulsar Client Feature Matrix - Rust.
If you need doc review, you can write docs on google doc and I would like to help review. And then you can publish the docs to https://docs.rs/pulsar/2.0.1/pulsar/ after the contents are finalized.
