Get subscription returning empty arrays

[MandarJKulkarni]

I have setup kafkabridge locally as well as in a kubernetes cluser.

Subscribing to topics is successful with POST API with response code 204.

curl --location --request POST 'http://localhost:8080/consumers/my-group/instances/my-consumer/subscription' \
--header 'content-type: application/vnd.kafka.v2+json' \
--data-raw '{
    "topics": [
        "mytopic1",
        "mytopic2"
    ]
}'

But listing the topics and subscription details using GET API always returns empty arrays.

curl --location --request GET 'http://localhost:8080/consumers/my-group/instances/my-consumer/subscription' \
--header 'content-type: application/vnd.kafka.v2+json'

response is

{
    "topics": [],
    "partitions": []
}

During debugging, the log reflects correct values in listSubscriptions method.

<idgeEndpoint:214> [oop-thread-2] Listing subscribed topics [SinkTopicSubscription(topic=mytopic1,partition=null,offset=null), SinkTopicSubscription(topic=mytopic2,partition=null,offset=null)]

But the API response is always empty.

[Anthuang]

I'm seeing this exact same issue as well.

[scholzj]

Are you consuming from the subscription? I get the empty arrays before I consume (probably because the consumer is not intialized at that point). But once I start consuming I get the answer with the right values. (tested with 0.21.4)

Not sure if that is intentional or not. @ppatierno might know.

[ppatierno]

Yeah, can you verify that you are able to consume by using the consumer endpoint and not the subscription one? Otherwise it could be a bug just on the subscription endpoint.

[MandarJKulkarni]

I am using the consumers endpoint. http://localhost:8080/consumers/test-consumer-group/instances/test-consumer/subscription

I am trying through postman directly. When I debug the kafka-bridge, I see the topic name and partitions correctly, but response doesn't have them. If it's working for you, can you please share the steps that you followed @scholzj ? I tried starting the consumer first and then the kafka-bridge application and vice versa too. But the response is always empty.

[scholzj]

I'm not sure I follow this. I didn't used any special steps -> just the curl commands from the docs.

[MandarJKulkarni]

Ok. I haven't tried curls, but the rest calls using Postman app only, which should give the same results I believe.

[scholzj]

Well, it is only abotu the actual HTTP query -> so any HTTP client should do.

[efiacor]

Hi @scholzj , is this a bug? Should we be able to see the subscriber info before actually consuming from the topics? I have replicated this behaviour locally myself. Only once the topic has records and the consumer has consumed do you see any subscriber info related the topics and partitions.

[scholzj]

I think it is expected. But @ppatierno might be able to give more authoritative answer.

[efiacor]

Hi @ppatierno , could you confirm the above behaviour is expected? Thanks

[ppatierno]

When you POST on /subscription endpoint, the bridge calls the subscribe on the Kafka consumer which just saves topics to subscribe internally. The real request to subscribe so joining a group happens when the first poll is done (so when you consume for the first time) even if it returns no records. Only that time the consumer is really subscribed because partitions were assigned to it after a rebalance. After a first consume, even with no records, you should see subscription via the GET on /subscription endpoint which calls the assignment on the underlying Kafka consumer. I would say that what you see is the normal behaviour of Kafka client if you try to do the same with a pure Java Kafka client and no bridge in the picture.

[ppatierno]

I tried again locally and subscriptions are showed after the consume not right after subscribe because of the underlying native Java Kafka client works as explained before. Going to close this one, if it's still an issue for you feel free to reopen.