kafka-connect-mq-source
kafka-connect-mq-source copied to clipboard
ibm-messaging
IBM MQ Admin reports too many connections when IBM MQ source connector is used
We have used IBM MQ source connector to push our data to kafka topics. Connector plugin is hosted in 2 of the Kafka connect instances present as part of our Kafka cluster.
For the configured Queue manager and channel name, we just have one connector in the Kafka cluster. But in IBM MQ admin we see 120+ connections/instances created for these 2 connect servers.
Because of these many connections, connector fails and is not able to process any of the messages. Can you kindly help with the solution for this issue?
Hello Team,
Is there any inputs on this issue ? Also, one more behavior what we have observed is whenever there is deployments and restarts of connect instances on our connect cluster, we could see count of connections against our MQ is raising. Is there any way to limit connections on MQ or kafka connector side ?
Thanks a lot, Rashmi
If you follow the instructions here to make GET /connectors/<connector-name>and GET /connectors/<connector-name>/tasks REST calls then you can see what Kafka Connect is doing. You can also check the logs of the Connect pods to see if there is any error output from the MQ connector.
Hi Dave, Thanks for your inputs. We have checked these REST call and it gives the task. This connector has just 1 task. State of the task is running. In connect log we do not see any error related to connections. But from mq side, there were 450 active connections and we had to restart mq channel to reduce or limit the connections. What could have been the reason for these many active connection on mq side ?
Thanks a lot, Rashmi
If you need to debug the connector then you can enable debug-level logging by setting the line log4j.logger.org.apache.kafka.clients.producer=DEBUG in connect-log4j.properties. Be aware that this will create very verbose log files, but it should allow you to identify what is causing the large number of connections.
Hello Team, We have enabled the logger on connector side. Still couldn't get much clue. Within the connector, this ibm mq client is being used- com.ibm.mq.allclient-9.1.5.0.jar. Is there any way to enable logging from this client ? If yes, could you please help us on how to configure this inside our connect-log4j.properties. Thanks!
Regards, Rashmi
late on this game.. we use the same connector and i don't see this behavior. At the MQ side number of readers in queue are equal to the number of tasks set in connector json. Does all these connections coming from same IP or 2 of these connect instance ?
Hello, we have the same issue.
We have a Kafka Connect cluster with 3 nodes. The connectors have 1 taks but we get 6 connection from the MQ side , after some time the connections increases and we need to restart MQ channel to reduce or limit the connections.
Connector Version: v1.3.1
@rashmidelampady did you solved the issue? I appreciate any advice on this.
Thanks Jorge
Hello Jorge, No we couldn't solve the issue. So we have shifted to enterprise confluent ibm mq connector which is stable.
On Thu, Jan 19, 2023 at 7:17 AM jahtux @.***> wrote:
Hello, we have the same issue.
We have a Kafka Connect cluster with 3 nodes. The connectors have 1 taks but we get 6 connection in the MQ said, after some time the connections increases and we need to restart MQ channel to reduce or limit the connections.
Connector Version: v1.3.1 https://github.com/ibm-messaging/kafka-connect-mq-source/releases/tag/v1.3.1
@rashmidelampady https://github.com/rashmidelampady did you solved the issue? I appreciate any advice on this.
Thanks Jorge
— Reply to this email directly, view it on GitHub https://github.com/ibm-messaging/kafka-connect-mq-source/issues/90#issuecomment-1396336681, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF7EGPXI55WUGEJGXLASZPDWTCMMZANCNFSM5BMYOANQ . You are receiving this because you were mentioned.Message ID: @.***>
Any updates? Look like there are several isssues related this "sticky MQ connection" one. I requested an IBM support PMR to be openned, this issue has a major impact as it "steals" MQ messages from the sourced queue until we force the whole Kafka Connect cluster to shutdown.
We actively looking into the issue. we will provide an update as soon as we are able to resolve it.
