Every brod_group_subscriber_v2 needs its own brod client

[pinkerltm]

I have three consumer groups, one should write messages just plain to the console, another one should update a state representing API with certain messages, and the third one should store every message into a long term database (crate). I use a supervisor to start three according brod_group_subscriber_v2 (see this GIST). If I also start three brod clients first and attach each group subscriber its own client, everything works perfectly so that Offsets are commited to Kafka for every group and reads start from the latest commited offset.

If I use only one client (as I thought it is possible), only the last group in my CHILD_SPEC works, both other do not receive handle_message calls.

At the moment starting a client for every group is not an issue for me, but later in our project we plan to use some tens of consumer groups, and I don't really think that it might be a good idea to run 20 to 30 brod clients and blocking ressources for each of them.

[k32]

Hi!

First of all, a disclaimer: group_subscriber_v2 behavior is currently experimental, there are some known issues with it that I, unfortunately, didn't have time to fix. In particular, it may not behave properly if your callback module crashes. As for your question: you should absolutely be able to use one brod client with multiple group_subscriber_v2's. I will look through it.

[k32]

Hi! Sorry for the long delay in responses. I think I identified the problem. It's likely related to how brod_topic_subscriber works internally. It tries to reuse a consumer, which is wrong.

[pinkerltm]

I have to admit I am surprised that the problem is indeed what you described and not that I have misused group_subscriber_v2 somehow ;) ... Do you think we should switch over to v1 for our case? Would that work in our setting?

[k32]

Yes, definitely switch to v1 for now, if you're doing anything serious. edit: P.S. Old behavior doesn't use topic_subscriber under the hood, hence it doesn't demonstrate this bug.

[pinkerltm]

ok, thanks, if that works, that would solve this issue for now....should I close it, or keep it open to track changes on v2?

[k32]

Please keep the issue open, I will use it as a reference for the fix.

[k32]

Hmm, turns out I was mistaken and v1 subscriber might be also affected by this issue.

[k32]

Sorry, it took me a while to fix it, mostly due to test suite deficiencies.

[gugahoa]

Hi all, I'm studying creating a parallel-consumer client for Elixir, and while experimenting with brod I ran into this issue. From the looks of it, the pr from @k32 didn't go through as they were hesitant regarding how it was implemented.

Is there any interesting in fixing this issue? If so I'm planning on taking it up, but would appreciate any guidance if possible

[jeffutter]

This issue is pretty old but seems to still be present. Anything we can do to help move a fix along?