jmx_exporter icon indicating copy to clipboard operation
jmx_exporter copied to clipboard

Config file reload concurrency when invoked via HTTP server

Open robinp-tw opened this issue 5 years ago • 3 comments

The HTTP server https://github.com/prometheus/client_java/blob/master/simpleclient_httpserver/src/main/java/io/prometheus/client/exporter/HTTPServer.java runs a pool of 5 serving threads, so metric collection can be invoked concurrently, or at least from different threads.

The design already seems to reflect this (see ConcurrentHashMap in the JmxMBeanPropertyCache), though some explicit remarks would be nice to have.

It seems config, configFile are not accessed synchronized, so they might be stale in different threads. On a quick eyeballing, it should still reload the config file fine per-thread, but why risk the ambiguity.

robinp-tw avatar May 12 '20 10:05 robinp-tw

I'm confused, are you reporting a bug in our concurrency handling and if so can you state exactly where the race is?

brian-brazil avatar May 12 '20 10:05 brian-brazil

Ah, sorry, was too entrenched in the context and omitted it. If I read it correctly, it's the collect method which can be called concurrently via the pooled http server. There, access to config field (via read of the lastUpdate and setting by reloadConfig) is not synchronized, which can lead to visibility issues between threads.

See https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L457

robinp-tw avatar May 13 '20 14:05 robinp-tw

Yeah, that looks like a race alright.

brian-brazil avatar May 13 '20 14:05 brian-brazil