open-telemetry
Thread and Thread pool metric automatic instrumentation
Is your feature request related to a problem? Please describe.
Getting thread metrics is pretty important, and is supported by most APMs. Before I can get all of my service owners to switch off of whatever APM they're currently using, I need at least a reasonable degree of feature parity in the metrics.
Describe the solution you'd like
Thread metrics available to export to otel and/or serve via prometheus. Including
- number of current threads total
- number of threads used out of a thread pool
- number of runnables executed by that pool
- pool queue size
Describe alternatives you've considered
No response
Additional context
No response
I have implemented the collection of thread pool metrics in my forked repository. including the following metrics:
- thread_pool_core_pool_size
- thread_pool_max_pool_size
- thread_pool_active_thread_count
- thread_pool_current_thread_count
- thread_pool_max_thread_count
- thread_pool_scheduled_task_count
- thread_pool_completed_task_count
- thread_pool_rejected_task_count
- thread_pool_queue_size labels:
- thread.name.pattern(e.g. http-nio-8080-*)
- thread.pool.class.name(e.g. java.util.concurrent.ThreadPoolExecutor)
if this is helpful, i'd like to create a pr
I think this is necessary, but the name of metrics needs to be considered. cc @trask @laurit @steverao
It looks like reasonable feature, @open-telemetry/java-instrumentation-approvers WDYT?
I think it is a reasonable request, but it is too vague. It would be helpful to know exactly which thread pools the users are interested it. It should already be possible to monitor many thread pools using https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent though users may need to write the configuration themself.
I think it is a reasonable request, but it is too vague. It would be helpful to know exactly which thread pools the users are interested it. It should already be possible to monitor many thread pools using https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent though users may need to write the configuration themself.
Most user-defined thread pools cannot be monitored through JMX metrics. And why do you think it si too vague, you don't know which thread pool should be monitored?
And why do you think it si too vague, you don't know which thread pool should be monitored?
No, I don't. There are many ways one could implement a thread pool, there won't be a catch all solution. Even something seemingly simple, like instrumenting ThreadPoolExecutor, immediately gets complicated when you consider that you somehow need to identify which ThreadPoolExecutor the metrics belong to.
No, I don't. There are many ways one could implement a thread pool
Indeed, there are many implementations of thread pools. However, it is sufficient to monitor the mainstream thread pools. Similar to the existing monitoring of connection pools, there are also various implementations of connection pools, but we only monitor a part of them, such as c3p0, dbcp, and so on.
Even something seemingly simple, like instrumenting
ThreadPoolExecutor, immediately gets complicated when you consider that you somehow need to identify whichThreadPoolExecutorthe metrics belong to.
I considered this issue as well. When I added two dimensions to the metrics related to thread pools, namely thread.name.pattern (e.g., http-nio-8080-*) and thread.pool.create.stack (indicating the method stack that created this thread pool), the association between a metric and its thread pool became clear.
@laurit Hi laurit，What do you think about my opinion？I do think is't a useful feature