spring-data-redis Registering stream consumers automatically with annotations

Basically what spring for kafka does, we would have an annotation like @RedisStreamListener to which we could provide the stream name and maybe some other options, you get the idea.

Are there maybe some limitations which make this not possible?

Mar 16 '23 16:03 NerminKarapandzic

Hi @NerminKarapandzic - Thank you for your interests and submitting a request on this topic.

I don't think this is the first time users have requested functionality, similar to what you are asking for here, before. See Issue #1004.

While I think this would be valuable and convenient add to Spring Data Redis, it also requires more deliberate and careful consideration. We would be happy to collaborate on something if you were willing to make a contribution. Otherwise, we will need to take this into considerations based on other priorities (deliverables) and interest.

However, in the meantime, if I may. I think you could achieve something like this already, if you were to combine Spring Data Redis with Spring Integration and possibly Spring Cloud Stream, for instance.

Spring Integration already contains nice support for inbound/outbound channel, Redis adapters (see doc). In addition, this might be able to be combined with Spring Cloud Stream [Applications]; see here (under "Available Applications").

I understand that this alternate approach would necessarily involve multiple dependencies, and that it would be more convenient to have something in Spring Data Redis specifically. But there is a reason these other libraries (like Spring Integration, like Spring Cloud Stream [Applications]) exist so we must be careful not to duplicate effort, which could be confusing to users as well.

Food for thought.

Mar 20 '23 06:03 jxblum

Thank you for the answer @jxblum, I would be happy to contribute. What would be the following steps?

Mar 20 '23 15:03 NerminKarapandzic

Hi @NerminKarapandzic - We try to make it as easy for our awesome community, like yourself, as possible to contribute to any Spring project, including Spring Data Redis.

We have documented the Spring (Data) contribution guidelines here. Basically, you will need to sign the CLA and submit a Pull Request (PR) with the changes that implement your idea and solution.

The PR will be picked up by the team for review and feedback.

Having said this, we had a Spring Data team triage call this morning and we also feel like it would be valuable to get feedback from the Spring Integration team and possibly the Spring Cloud Stream team to coordinate our efforts there as well (again, to avoid duplicate efforts and overlapping functionality that could cause confusion for our other users). We can coordinate and collect feedback from the Spring Integration/Spring Cloud Stream teams internally and I can summarize in the comments here.

In terms of priorities, I am currently in the middle of a few other tickets that need my attention. I will have those wrapped up in a week or two (I hope, barring any interruptions).

Looking forward to collaborating with you.

Mar 20 '23 15:03 jxblum

Great, @jxblum . I will already start looking into it and think/work on some implementation, but do mention me here please when you have other feedback from your meetings.

Mar 20 '23 19:03 NerminKarapandzic

Thank you @NerminKarapandzic.

My advice would be to maybe start with just a simple prototype at first, a test and maybe an example, that would answer the question, "What would this look and feel like to users?", or "How would they use it (for different use cases, in different scenarios/contexts/etc?)"

This might seem obvious to use, but what about someone new to Spring specifically, Spring Data Redis? I know you mentioned Spring Kafka, but with a Spring/Redis spin on it. RedisStreamListener is not bad, but we can maybe play around here more as well. Include support or filtering, maybe conversion or transformation of elements arriving from the stream, or events coming from some source, etc.

I would not try to solve this entire problem by yourself since @mp911de was also thinking of tying this feature together with Redis pub/sub (doc), and Spring Data Redis's RedisMessageListenerContainer (Javadoc), and maybe few other areas as well.

Again, we will definitely want to get some feedback from the Spring Integration and Spring Cloud Stream team.

Thanks again.

Mar 20 '23 21:03 jxblum

Hey @jxblum, I came up with some basic version and a test to prove it. I would like to get some feedback and ideas before I continue working on it. Don't mind the commit messages and coding style yet. Here's where you can check the current work: https://github.com/spring-projects/spring-data-redis/compare/main...NerminKarapandzic:spring-data-redis:issue/DATAREDIS-2528

Mar 28 '23 17:03 NerminKarapandzic

Thank you so much @NerminKarapandzic! Really appreciate this. I will have a look in a few days. Currently, involved in few other activities that need my quick attention. Will get back to you soon.

Mar 28 '23 18:03 jxblum

Hey @jxblum , just wondering if you had a chance to look at this?

Apr 13 '23 19:04 NerminKarapandzic

Hi @NerminKarapandzic - Sorry for the delayed response. No, sorry. This ticket is not deemed high priority at the moment, and needs better cross-project coordination (Spring Integration and Spring Cloud Stream teams) for sure.

This will take a bit of time. Currently, the entire team is preparing for critical GA release coming up in (mid) May (see May in Spring Release Calendar).

I promise to get back to you on this.

Apr 26 '23 16:04 jxblum