components-contrib
components-contrib copied to clipboard
dapr
[Proposal] Create an adapter to use PubSub components as Bindings
Currently, there are several bindings that have a pubsub counterpart (E.g. Kafka, Rabbit, Pulsar, NATS, Redis). In many cases the code in the PubSub component is far more robust. Given the similarity between bindings (Data + Metadata) and pubsub (Topic + Data + Metadata), an adapter could be used that specifies the read/invoke topics to subscribe and publish to. The primary motivation is that adapter reuses code and users can expect the same behavior between the binding and pubsub.
Here are some issues that would possibly be resolved for Kafka:
#1277 #1084 #955 #1006 #168
The downside is that if there are current inconsistencies in the component settings, then migrating to this approach would break existing versions. Since none of these components are marked as stable (yet), now is the time to do this if at all.
It is a design issue as bindings are more abstract than pubsub, we should better guide users to choose the most suitable one instead of making the two more obscure.
/cc @artursouza @yaron2
It is a design issue as bindings are more abstract than pubsub, we should better guide users to choose the most suitable one instead of making the two more obscure.
Hi, @daixiang0! I don't disagree with this statement. I have opinions about this duality as well; however, users are already using the binding versions of pubsub components and experiencing issues that are not present in the pubsub component. This proposal isn't intended to debate the merit of bindings for message brokers. Instead, this proposal aims to remove redundant or inferior code in favor of reusing the more robust pubsub code. If anything it provides a simpler, more maintainable option for developers to subscribe/publish to topics without having to configure or have knowledge of pubsub/topics inside their applications.
Might this also be improved by simply refactoring pubsub and binding components to use shared code like the Redis components? That would allow some foundational aspects like service connectivity, authentication, etc... to be used in both even if the functionality diverges.
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.
I have a question about the design principle of pubsub and binding:
Since we have pubsub components for these message broker, why we still provide binding components for them?
Do we provide more capabilities than pubsub in binding for these message broker? For this question I checked kafka binding and kafka pubsub, the answer is NO.
And I checked carefully for the code of kafka binding and kafka pubsub.
output binding
In kafka output binding, only "create" operation is supported, and the functionality is totally the same as publish() in kafka pubsub, while the implementation code of publish() in kafka pubsub is better, or more careful to handle details: 1. check if component is closed. 2. support more metadata, not only "partitionKey".
input binding
The implementation code in input binding Read() method is very similar to the Subscribe() method of pubsub. The biggest difference is that there is a retry logic in pubsub. This retry logic is also in the ConsumeClaim() of ConsumerGroupHandler.
So the question is that why we need retry in kafka subscribe but not need in kafka input binding? It seems the reason is due to the code implementation of kafka binding.
My conclusion about kafka binding is that it can be replaced with kafka pubsub from the point of view of functionality.
Discuss
I list some possible solution to discuss:
- Proposal 1 : Remove binding components for message brokers, since their pubsub components are enough.
- Proposal 2 : Keep binding components for message brokers, add adapters to the pubsub components and reuse the code in component level. As @pkedy suggested in this issue.
- Proposal 3 : Keep binding components for message brokers, extract the common code of kafka and reuse them in both kafka binding and kafka pubsub. In fact, for redis, we have already do this code reuse. Please have a look at
internal/component/redis - Proposal 4 : this is mixed proposal —— take proposal 2 first, and mark the kafka binding components as "deprecated", later take proposal 1 to delete kafak binding.
Some background: I'm taking the task to add certification test for kafka binding, yet I found that to do this I have to repeat all the work of certification test for kafka pubsub. I don't want to simply control + c && control + v. This is not a good choice. I would like to do this certification after we discussed and made decision.
@pkedy: thank for your good suggestion about this issue.
We should not remove the bindings that have pub/sub equivalents, specifically Kafka, because they allow users to contribute and use more specialized features from the underlying system. For example, a Kafka binding can expose an API that returns the number of messages in a topic.
The fact it's not there today does not mean we should block this functionality from being added at any point in the future by removing the component.
That is a great point, @yaron2. Does it mean we should not simply make the Kafka binding a wrap of the pubsub component and keep those as completely separate implementations?
We can use the suggested adopter pattern to reuse what code we can (if at all, but I believe most code can be reused).
As long as we allow bindings to add operations (which is their goal for flexibility) and their own unique metadata, we can refactor the code however we see fit.
We can use the suggested adopter pattern to reuse what code we can (if at all, but I believe most code can be reused).
As long as we allow bindings to add operations (which is their goal for flexibility) and their own unique metadata, we can refactor the code however we see fit.
Hi, yaron, let me try to do a code refactory for kafka pubsub and binding, I will extract some common code to reuse, and use adapter for publish (outpub binding create operation) & subscribe (input binding read method).
@artursouza @yaron2 @pkedy I have submitted the first version of the code refactory for kafka pubsub component and kafka binding component accrording to above discussion.
I split this code refactory into two parts:
-
refactory kafka pubsub compoent to extract the common kafka code
These common kafka code is moved to
internal/kafka.Kafka pubsub compoent now is based on this common code and it adapters its pubsub api method to the common code like init() / publish() / subscribe() / close().
This refactory should be safe since it only split and move the code, no functionality change.
-
refactory kafka binding component to reuse the common kafka code
This is a BIG change to kafka binding component since now it's code base is moved to the common kafka code extracting from kafka pubsub component.
The benefit is that now kafka component is have the same features and implementations as kafka pubsub component.
The challenge is that we need to careful handle the difference.
The second PR should be reviewed very carefully.
I suggest we keep the adapted version as kafka binding v2 and keep the old version as-is in v1. This way, we don't break existing apps. Even as alpha component, kafka is a popular broker.
Do we know of breaking changes here?
Yes. The E2E tests are failing already because of that in the PR https://github.com/dapr/dapr/actions/runs/2243065335