JMX metrics collection is not working (e.g. Hazelcast metrics about maps)

[ghilainm]

Describe the bug

Using the latest version of elastic apm agent (1.36), I can't collect JMX metrics about Hazelcast maps.

2023-02-06 14:39:59,140 [hz.cool_zhukovsky.cached.thread-16] WARN co.elastic.apm.agent.jmx.JmxMetricTracker - Can't create metric 'xxx' because attribute 'hits' could not be found.

The problem is created because Hazelcast populates MBean attributes on the fly (see com.hazelcast.internal.metrics.jmx.MetricsMBean). The metrics are not yet ready when the MBeanServerNotification event is fired.

public class MetricsMBean implements DynamicMBean {
private final ConcurrentMap<String, TriTuple<String, AtomicReference<Number>, Type>> metrics = new ConcurrentHashMap<>();

/**
 * Adds a metric if necessary and sets its value.
 */
void setMetricValue(String name, String unit, Number value, Type type) {
    TriTuple<String, AtomicReference<Number>, Type> metricTuple = metrics.get(name);
    if (metricTuple == null) {
        metricTuple = of(unit, new AtomicReference<>(), type);
        metrics.put(name, metricTuple);
    }
    metricTuple.element2.lazySet(value);
}

void removeMetric(String name) {
    metrics.remove(name);
}

@Override
public Object getAttribute(String attributeName) throws AttributeNotFoundException {
    TriTuple<String, AtomicReference<Number>, Type> attribute = metrics.get(attributeName);
    if (attribute == null) {
        throw new AttributeNotFoundException(attributeName);
    }
    return attribute.element2.get();
}

@Override
public AttributeList getAttributes(String[] attributes) {
    return Arrays.stream(attributes)
                 .filter(metrics::containsKey)
                 .map(a -> uncheckCall(() -> new Attribute(a, getAttribute(a))))
                 .collect(toCollection(AttributeList::new));
}

}

Other metrics can be properly collected, therefore it is not a problem of agent connectivity or such.

Steps to reproduce

• Configure metrics export: ELASTIC_APM_CAPTURE_JMX_METRICS=object_name[com.hazelcast:type=Metrics,*,prefix=map,tag0="name=MyMap"] attribute[hits:metric_name=MyMap_hits]
• Create an instance of Hazelcast
• Create a map called MyMap
• Add stuff to the map
• See that the warning is produced in the logs

Expected behaviour

Metrics are reported to Elastic.

Possible solution:

• Implement some kind exponential back off and try again registering JMXMetrics later on.

Improvements:

• Agent should maybe also listen to AttributeChangeNotification event in addition to MBeanServerNotification (this wouldn't solve the issue of Hazelcast as they don't implement such notification).

[eyalkoren]

I am not sure why the warning says attribute 'getCount' could not be found if your configuration looks for an attribute hits, so I will assume that these are only non-corresponding examples.

Possible solution:

• Implement some kind exponential back off and try again registering JMXMetrics later on.

I don't see us prioritizing this soon, but maybe you can implement that by relying on the fact that the capture_jmx_metrics config option is dynamic: when you know it's time, update the value of this config option (e.g. through the elastic.apm.capture_jmx_metrics system property) to capture the dynamically added attributes.

Improvements:

• Agent should maybe also listen to AttributeChangeNotification event in addition to MBeanServerNotification (this wouldn't solve the issue of Hazelcast as they don't implement such notification).

This makes sense. If we do, we can cache those configurations for which the attributes were not found. Anyway, since it won't help in your case, we'll not look into that now.

[ghilainm]

You are right, mismatch in the example :).

Just setting System.setProperty("elastic.apm.capture_jmx_metrics") would force the agent to reload its configuration?

[eyalkoren]

Just setting System.setProperty("elastic.apm.capture_jmx_metrics") would force the agent to reload its configuration?

The agent polls the system properties every 30 seconds and if a value changes it triggers anything that listens on such changes, like the JmxMetricTracker.