Add additional metrics provided by .NET Core runtime and kestrel web server

[SebZal]

Starting in .NET Core 3.0 basic diagnostic information are sent from the .NET Core runtime and some libraries. We are interested in collecting metrics from the following counters:

"System.Runtime" counters

• time-in-gc
• alloc-rate
• cpu-usage
• exception-count
• gc-heap-size
• gen-0-gc-count
• gen-0-size
• gen-1-gc-count
• gen-1-size
• gen-2-gc-count
• gen-2-size
• loh-size
• poh-size
• gc-fragmentation
• monitor-lock-contention-count
• active-timer-count
• assembly-count
• threadpool-completed-items-count
• threadpool-queue-length
• threadpool-thread-count
• working-set
• il-bytes-jitted
• method-jitted-count

Microsoft-AspNetCore-Server-Kestrel

• connection-queue-length
• connections-per-second
• current-connections
• current-tls-handshakes
• current-upgraded-requests
• failed-tls-handshakes
• request-queue-length
• tls-handshakes-per-second
• total-connections
• total-tls-handshakes

Would it be possible to extend the APM agent with these metrics? Some of these are already provided through existing metric providers in the APM .NET Agent and could probably be omitted. I could create a PR with these changes if this sounds reasonable.

This could be solved by creating two additional providers, e.g. SystemRuntimeMetricsProvider and KestrelMetricsProvider. Both of these could be similarly implemented as in the GcMetricsProvider where System.Diagnostics.Tracing.EventListener is used in order to listen to metrics.

[russcam]

Thanks for opening, @SebZal - I've added a note to have a chat about these in our weekly sync

[SebZal]

I have added a suggested solution here: https://github.com/elastic/apm-agent-dotnet/pull/1076

[stefannikolei]

Is there any news on this? I think it should be possible to have all of this in the .NET5 world and beyond and keep the old behaviour for :net core 3.1 and before

[russcam]

Hi @stefannikolei, no new news to report; haven't forgotten about this, but needing to prioritize other items at present.

We've been talking recently about the ability to send custom metrics (https://github.com/elastic/apm-agent-dotnet/issues/322), so I think we would look at this at the same time.

[TailyFair]

Any updates regarding this? I struggle to find proper solutions to visualize .NET runtime metrics on ELK stack. Even AppMetrics nuget is no longer reporting GC metrics since .NET 5 incompatabilities.

[solomon-okolie]

When can we get an update about this issue?

[nixxcz]

Any news about this? I would really appreciate more detailed metrics than basic CPU/memory utilization.