KnowStreaming icon indicating copy to clipboard operation
KnowStreaming copied to clipboard

getAttributes运行时错误

Open Huyueeer opened this issue 4 years ago • 5 comments

日志文件中暴露大量关于c.x.k.manager.service.service.impl.JmxServiceImpl - get attributes failed,的错误,具体内容如下: 日志内容,一天可能会打60多个log image 然后看具体报错:

2021-09-21 23:51:18.698 [Collect-Metrics-Thread-1-91] ERROR c.x.k.manager.service.service.impl.JmxServiceImpl - get attributes failed, metrics:TopicMetrics{appId='null', clusterId=8, topicName='CDC-IMX-CTO.dbo.INI', metricsMap={}} objectName:kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=CDC-IMX-CTO.dbo.INI.
java.io.IOException: Connection closed
	at javax.management.remote.rmi.RMIConnector.getMBeanServerConnection(RMIConnector.java:400)
	at javax.management.remote.rmi.RMIConnector.getMBeanServerConnection(RMIConnector.java:389)
	at com.xiaojukeji.kafka.manager.common.utils.jmx.JmxConnectorWrap.getAttributes(JmxConnectorWrap.java:126)
	at com.xiaojukeji.kafka.manager.service.service.impl.JmxServiceImpl.getAndSupplyAttributes2BaseMetrics(JmxServiceImpl.java:263)
	at com.xiaojukeji.kafka.manager.service.service.impl.JmxServiceImpl.getTopicMetrics(JmxServiceImpl.java:171)
	at com.xiaojukeji.kafka.manager.service.service.impl.JmxServiceImpl.getTopicMetrics(JmxServiceImpl.java:130)
	at com.xiaojukeji.kafka.manager.service.service.impl.JmxServiceImpl$1.call(JmxServiceImpl.java:93)
	at com.xiaojukeji.kafka.manager.service.service.impl.JmxServiceImpl$1.call(JmxServiceImpl.java:90)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

我在源码中找到了这段代码:

public AttributeList getAttributes(ObjectName name, String[] attributes) throws
            MBeanException,
            AttributeNotFoundException,
            InstanceNotFoundException,
            ReflectionException,
            IOException {
        try {
            acquire();
            MBeanServerConnection mBeanServerConnection = jmxConnector.getMBeanServerConnection();
            return mBeanServerConnection.getAttributes(name, attributes);
        } finally {
            atomicInteger.incrementAndGet();
        }
    }

现在执行到MBeanServerConnection mBeanServerConnection = jmxConnector.getMBeanServerConnection();就try到的错误,所以是否可以说jmx是连接成功的,但是在getMBeanServerConnection()环节出现了错误,不是很明白是什么原因。

Huyueeer avatar Sep 22 '21 02:09 Huyueeer

好的,这个问题我们近期关注一下~

ZHAOYINRUI avatar Nov 10 '21 06:11 ZHAOYINRUI

已在最新2.5.0版本修复

ZQKC avatar Jan 07 '22 04:01 ZQKC

@ZQKC 关于get attributes failed的错误还是在继续存在,我发现一些规律,这些Topic都是CDC,或者自动生成的一些Topic,是不是对这种类型的Topic采集信息有什么问题?,包括都不存在appId。 image image

另外,这部分是否和kafka开放指标有关系?

Huyueeer avatar Jan 25 '22 02:01 Huyueeer

辛苦提供一下异常的线程堆栈,看一下是怎么样触发的

ZQKC avatar Jan 26 '22 01:01 ZQKC

@ZQKC 看一下这个,和上一个版本的报错差不多。 image

Huyueeer avatar Jan 26 '22 05:01 Huyueeer

beta.3版本已进行优化。 出现问题的原因是连接被关闭了,当前处理方式是,发现连接被关闭的异常后,会对JMX连接进行重建。因此该错误还是会出现,但是出现后会进行重建修复~

ZQKC avatar Sep 22 '22 11:09 ZQKC