Cooperation on Pulsar client for Scala

[Anonymitaet]

Hi @sksamuel,

Hope you’re having a great holiday.

My name is Yu Liu. I'm a Pulsar committer and technical writer.

When browsing the Pulsar website, I found that you have developed a wonderful Scala client for Pulsar and draw much attention. Thanks for this great work!

Since good projects are collaborations between motivated individuals, I'm wondering if you'd like to donate the Scala client to Apache Pulsar, which is beneficial for attracting more programmers to share quality code, appealing to more technical writers to develop docs (I found you've written some guides and I'd like to contribute together), and allowing more users to use it conveniently.

Any thoughts on this? Looking forward to hearing from you.

Merry Christmas with lots of love.

Yu

[gmethvin]

I assume you're asking for us to donate the project to the ASF. Would there be any additional changes needed?

Ideally we'd want to maintain our own release schedule and ability to make API changes when necessary. This is still a fairly rapidly evolving project.

We might want to think about simplifying the API a bit to minimize the breaking changes we need to make on new versions. I'm also not sure if an official library should attempt to support every effect type and JSON library in wide use; those might be better off as separate unofficial projects.

[sksamuel]

The advantages of being with Apache are the "official" status it confers. I have no experience with Apache so my worry is that things like the release process become bogged down in "red tape". Right now we can do a release by clicking a button with github actions. Or if we want to change something, Greg and I can decide and implement it as quickly as we want.

Would a relatively niche library gain much from Apache status ?

[Anonymitaet]

Can any of you help answer these questions? @sijie @jiazhai @codelipenghui

[Anonymitaet]

Hi @gmethvin and @sksamuel

I found that @sijie has answered some similar questions elsewhere, so I copied his feedback here. Hope this helps.

By contributing this project to the ASF, it means:

1. transfer the project under https://github.com/apache. so the Pulsar PMC will be responsible for this repo.
2. We need to clean up the license headers to make sure the source code is aligned with ASF policy.
3. We need to make sure the release process for the project is following the ASF policy.

ASF policy doesn't enforce any rules on coding guidelines or code structure. It is typically managed by the committers and contributors themselves once they agree on a certain guideline.

Upon contribution, the company currently owns a client repository has to sign a software grant agreement to the ASF and release the software to the ASF. You will still be the main contributor maintaining and developing the project and attract more people to contribute to that project.

Hope this clarifies your questions. If this is something you are interested in doing, we can help you go through the process.

[sijie]

I think it will probably slow down a bit on the release process given the new release has to follow the ASF policy.

An alternative way to maintain the momentum of this project and give more exposure to this scala client is to maintain a documentation page about this scala client on the pulsar website. Does that sound like a good starting approach? @gmethvin @sksamuel

@Anonymitaet does that resolve the documentation concern you have?

[sksamuel]

I think the documentation idea is an excellent one to start. @gmethvin ?

[Anonymitaet]

Hi @sijie, as instructed by @codelipenghui, we will start working on the Scala user guide and publish it on the Pulsar website after the Scala client is donated to Pulsar. I've read the Scala docs and of course I can start working on it along with the process of donation if needed.

[sijie]

@Anonymitaet Let's not worry about the donation process. It is okay to add a page about the scala client. We just need to note that the client is hosted in a different repo.

[gmethvin]

Yes, I think the documentation would be a good place to start.

[gmethvin]

In the long run I think it'd be valuable to provide an official Scala library that supports basic functionality, but doesn't attempt to integrate with third-party libraries. In practice I suspect most people would need to use at least one of the integration libraries, but keeping those libraries separate would allow us to iterate faster.

Right now, the main changes in the core API have been to support new Pulsar APIs, and to add additional configuration options. We might want to consider some API changes to improve backwards compatibility going forward, like using a builder pattern instead of a case class for configuration.

[Anonymitaet]

Hi @gmethvin and @sksamuel,

As we all agreed that the doc is a good place to start, which way do you prefer to contribute the doc?

#1: contribute the Scala user guide to the Pulsar website yourself (you can take Pulsar client docs and Pulsar client feature matrix as references) and then I'll review

or

#2: I draft one based on Scala readme, request your review and technical inputs (if any), and then submit doc PR to Pulsar repo

Let me know your thoughts. Thank you.

[Anonymitaet]

Hi @gmethvin and @sksamuel, any feedback on the suggestion?

[gmethvin]

So you'd like to have the pulsar4s docs look like the docs for the official clients, just with an additional note that it's a third-party library?

I'm happy to contribute to the docs directly, though I'm a little busy the next couple weeks so I'm not sure how soon I'd be able to send the PR.

Do you have any suggestions based on what you read in the readme, or your experience using the client? I'm sure there are things that could be added or clarified.

[Anonymitaet]

@gmethvin many thanks for your feedback.

So you'd like to have the pulsar4s docs look like the docs for the official clients, just with an additional note that it's a third-party library?

We can add pulsar4s docs on the Pulsar doc website and add a note that this is a third-party client in Scala.

Do you have any suggestions based on what you read in the readme, or your experience using the client? I'm sure there are things that could be added or clarified.

• I've created a Pulsar Client Doc Template which includes writing tips and Pulsar Client Feature Matrix. You can take them as references.
• Pulsar Client Feature Matrix is quite helpful for providing complete docs. We are collecting features supported by different clients and will make the contents public once we finish, so could you please help tick items for the Scala client? Thank you.

I'm happy to contribute to the docs directly, though I'm a little busy the next couple weeks so I'm not sure how soon I'd be able to send the PR.

As we know, quality docs help users use the project and be able to improve on the project in return. Hope we can work together to build a robust, mature, and successful project. Looking forward to hearing from you soon.

[Anonymitaet]

Hi @gmethvin, any feedback for the client docs and the Pulsar Client Feature Matrix? Thank you