Map the metrics of the OpenTelemetry Collector Host Metrics Receiver to ECS and make them visible in Elastic Infra Inventory view (see `metrics-` indices / datastreams)

[cyrille-leclerc]

Map the metrics of the OpenTelemetry Collector Host Metrics receiver to Elastic Infrastructure monitoring (ECS...) and store them in metrics- indices so that they are properly visualized in the Elastic Observability infrastructure view (aka wafflemap).

We probably want to store these metrics in metrics- indices so that they are transparently used by the infrastructure view (aka waffle map) that by default picks metrics-*,metricbeat-*.

Note that once we have mapped the metrics produced by the Host Metrics receiver, we will consider supporting the:

• Kubernetes Receiver or any more popular OpenTelemetry receiver used to monitor Kubernetes
• AWS Container Insights Receiver or any more popular OpenTelemetry receiver used to monitor containers running on AWS

Here are some metrics collected by the Host Metrics Receiver (v0.27) on a Mac and visualized on the Prometheus exporter page. The cpu, disk, processes scrappers are not available on Mac.

# HELP system_cpu_load_average_15m Average CPU Load over 15 minutes.
# TYPE system_cpu_load_average_15m gauge
system_cpu_load_average_15m 3.51123046875
# HELP system_cpu_load_average_1m Average CPU Load over 1 minute.
# TYPE system_cpu_load_average_1m gauge
system_cpu_load_average_1m 2.92578125
# HELP system_cpu_load_average_5m Average CPU Load over 5 minutes.
# TYPE system_cpu_load_average_5m gauge
system_cpu_load_average_5m 3.158203125
# HELP system_filesystem_inodes_usage FileSystem inodes used.
# TYPE system_filesystem_inodes_usage gauge
system_filesystem_inodes_usage{device="/dev/disk1s1s1",mode="ro",mountpoint="/",state="free",type="apfs"} 4.881892887e+09
system_filesystem_inodes_usage{device="/dev/disk1s1s1",mode="ro",mountpoint="/",state="used",type="apfs"} 559993
system_filesystem_inodes_usage{device="/dev/disk1s2",mode="rw",mountpoint="/System/Volumes/Data",state="free",type="apfs"} 4.87977125e+09
system_filesystem_inodes_usage{device="/dev/disk1s2",mode="rw",mountpoint="/System/Volumes/Data",state="used",type="apfs"} 2.68163e+06
system_filesystem_inodes_usage{device="/dev/disk1s3",mode="rw",mountpoint="/System/Volumes/Preboot",state="free",type="apfs"} 4.882451373e+09
system_filesystem_inodes_usage{device="/dev/disk1s3",mode="rw",mountpoint="/System/Volumes/Preboot",state="used",type="apfs"} 1507
system_filesystem_inodes_usage{device="/dev/disk1s5",mode="rw",mountpoint="/System/Volumes/VM",state="free",type="apfs"} 4.882452877e+09
system_filesystem_inodes_usage{device="/dev/disk1s5",mode="rw",mountpoint="/System/Volumes/VM",state="used",type="apfs"} 3
system_filesystem_inodes_usage{device="/dev/disk1s6",mode="rw",mountpoint="/System/Volumes/Update",state="free",type="apfs"} 4.882452863e+09
system_filesystem_inodes_usage{device="/dev/disk1s6",mode="rw",mountpoint="/System/Volumes/Update",state="used",type="apfs"} 17
system_filesystem_inodes_usage{device="devfs",mode="rw",mountpoint="/dev",state="free",type="devfs"} 0
system_filesystem_inodes_usage{device="devfs",mode="rw",mountpoint="/dev",state="used",type="devfs"} 796
system_filesystem_inodes_usage{device="drivefs",mode="rw",mountpoint="/Volumes/GoogleDrive",state="free",type="dfsfuse_DFS"} 4.294967295e+09
system_filesystem_inodes_usage{device="drivefs",mode="rw",mountpoint="/Volumes/GoogleDrive",state="used",type="dfsfuse_DFS"} 0
system_filesystem_inodes_usage{device="map auto_home",mode="rw",mountpoint="/System/Volumes/Data/home",state="free",type="autofs"} 0
system_filesystem_inodes_usage{device="map auto_home",mode="rw",mountpoint="/System/Volumes/Data/home",state="used",type="autofs"} 0
# HELP system_filesystem_usage Filesystem bytes used.
# TYPE system_filesystem_usage gauge
system_filesystem_usage{device="/dev/disk1s1s1",mode="ro",mountpoint="/",state="free",type="apfs"} 7.388766208e+10
system_filesystem_usage{device="/dev/disk1s1s1",mode="ro",mountpoint="/",state="reserved",type="apfs"} 4.1072654336e+11
system_filesystem_usage{device="/dev/disk1s1s1",mode="ro",mountpoint="/",state="used",type="apfs"} 1.5348969472e+10
system_filesystem_usage{device="/dev/disk1s2",mode="rw",mountpoint="/System/Volumes/Data",state="free",type="apfs"} 7.388766208e+10
system_filesystem_usage{device="/dev/disk1s2",mode="rw",mountpoint="/System/Volumes/Data",state="reserved",type="apfs"} 1.97999616e+10
system_filesystem_usage{device="/dev/disk1s2",mode="rw",mountpoint="/System/Volumes/Data",state="used",type="apfs"} 4.06275551232e+11
system_filesystem_usage{device="/dev/disk1s3",mode="rw",mountpoint="/System/Volumes/Preboot",state="free",type="apfs"} 7.388766208e+10
system_filesystem_usage{device="/dev/disk1s3",mode="rw",mountpoint="/System/Volumes/Preboot",state="reserved",type="apfs"} 4.2563018752e+11
system_filesystem_usage{device="/dev/disk1s3",mode="rw",mountpoint="/System/Volumes/Preboot",state="used",type="apfs"} 4.45325312e+08
system_filesystem_usage{device="/dev/disk1s5",mode="rw",mountpoint="/System/Volumes/VM",state="free",type="apfs"} 7.388766208e+10
system_filesystem_usage{device="/dev/disk1s5",mode="rw",mountpoint="/System/Volumes/VM",state="reserved",type="apfs"} 4.22854152192e+11
system_filesystem_usage{device="/dev/disk1s5",mode="rw",mountpoint="/System/Volumes/VM",state="used",type="apfs"} 3.22136064e+09
system_filesystem_usage{device="/dev/disk1s6",mode="rw",mountpoint="/System/Volumes/Update",state="free",type="apfs"} 7.388766208e+10
system_filesystem_usage{device="/dev/disk1s6",mode="rw",mountpoint="/System/Volumes/Update",state="reserved",type="apfs"} 4.26072055808e+11
system_filesystem_usage{device="/dev/disk1s6",mode="rw",mountpoint="/System/Volumes/Update",state="used",type="apfs"} 3.457024e+06
system_filesystem_usage{device="devfs",mode="rw",mountpoint="/dev",state="free",type="devfs"} 0
system_filesystem_usage{device="devfs",mode="rw",mountpoint="/dev",state="reserved",type="devfs"} 0
system_filesystem_usage{device="devfs",mode="rw",mountpoint="/dev",state="used",type="devfs"} 235520
system_filesystem_usage{device="drivefs",mode="rw",mountpoint="/Volumes/GoogleDrive",state="free",type="dfsfuse_DFS"} 7.0193278976e+10
system_filesystem_usage{device="drivefs",mode="rw",mountpoint="/Volumes/GoogleDrive",state="reserved",type="dfsfuse_DFS"} 4.29769895936e+11
system_filesystem_usage{device="drivefs",mode="rw",mountpoint="/Volumes/GoogleDrive",state="used",type="dfsfuse_DFS"} 0
system_filesystem_usage{device="map auto_home",mode="rw",mountpoint="/System/Volumes/Data/home",state="free",type="autofs"} 0
system_filesystem_usage{device="map auto_home",mode="rw",mountpoint="/System/Volumes/Data/home",state="reserved",type="autofs"} 0
system_filesystem_usage{device="map auto_home",mode="rw",mountpoint="/System/Volumes/Data/home",state="used",type="autofs"} 0
# HELP system_memory_usage Bytes of memory in use.
# TYPE system_memory_usage gauge
system_memory_usage{state="free"} 2.21372416e+08
system_memory_usage{state="inactive"} 5.07748352e+09
system_memory_usage{state="used"} 1.1881013248e+10
# HELP system_network_connections The number of connections.
# TYPE system_network_connections gauge
system_network_connections{protocol="tcp",state="CLOSED"} 7
system_network_connections{protocol="tcp",state="CLOSE_WAIT"} 2
system_network_connections{protocol="tcp",state="CLOSING"} 0
system_network_connections{protocol="tcp",state="DELETE"} 0
system_network_connections{protocol="tcp",state="ESTABLISHED"} 303
system_network_connections{protocol="tcp",state="FIN_WAIT_1"} 0
system_network_connections{protocol="tcp",state="FIN_WAIT_2"} 0
system_network_connections{protocol="tcp",state="LAST_ACK"} 0
system_network_connections{protocol="tcp",state="LISTEN"} 24
system_network_connections{protocol="tcp",state="SYN_RECEIVED"} 0
system_network_connections{protocol="tcp",state="SYN_SENT"} 0
system_network_connections{protocol="tcp",state="TIME_WAIT"} 0
# HELP system_network_dropped The number of packets dropped.
# TYPE system_network_dropped counter
system_network_dropped{device="ap1",direction="receive"} 0
system_network_dropped{device="ap1",direction="transmit"} 0
system_network_dropped{device="awdl0",direction="receive"} 0
system_network_dropped{device="awdl0",direction="transmit"} 0
system_network_dropped{device="bridg",direction="receive"} 0
system_network_dropped{device="bridg",direction="transmit"} 0
system_network_dropped{device="en0",direction="receive"} 0
system_network_dropped{device="en0",direction="transmit"} 177
system_network_dropped{device="en1",direction="receive"} 0
system_network_dropped{device="en1",direction="transmit"} 0
system_network_dropped{device="en2",direction="receive"} 0
system_network_dropped{device="en2",direction="transmit"} 0
system_network_dropped{device="en3",direction="receive"} 0
system_network_dropped{device="en3",direction="transmit"} 0
system_network_dropped{device="en4",direction="receive"} 0
system_network_dropped{device="en4",direction="transmit"} 0
system_network_dropped{device="en5",direction="receive"} 0
system_network_dropped{device="en5",direction="transmit"} 69
system_network_dropped{device="gif0",direction="receive"} 0
system_network_dropped{device="gif0",direction="transmit"} 0
system_network_dropped{device="llw0",direction="receive"} 0
system_network_dropped{device="llw0",direction="transmit"} 0
system_network_dropped{device="lo0",direction="receive"} 0
system_network_dropped{device="lo0",direction="transmit"} 0
system_network_dropped{device="stf0",direction="receive"} 0
system_network_dropped{device="stf0",direction="transmit"} 0
system_network_dropped{device="utun0",direction="receive"} 0
system_network_dropped{device="utun0",direction="transmit"} 0
system_network_dropped{device="utun1",direction="receive"} 0
system_network_dropped{device="utun1",direction="transmit"} 0
# HELP system_network_errors The number of errors encountered.
# TYPE system_network_errors counter
system_network_errors{device="ap1",direction="receive"} 0
system_network_errors{device="ap1",direction="transmit"} 0
system_network_errors{device="awdl0",direction="receive"} 0
system_network_errors{device="awdl0",direction="transmit"} 0
system_network_errors{device="bridg",direction="receive"} 0
system_network_errors{device="bridg",direction="transmit"} 0
system_network_errors{device="en0",direction="receive"} 0
system_network_errors{device="en0",direction="transmit"} 2296
system_network_errors{device="en1",direction="receive"} 0
system_network_errors{device="en1",direction="transmit"} 0
system_network_errors{device="en2",direction="receive"} 0
system_network_errors{device="en2",direction="transmit"} 0
system_network_errors{device="en3",direction="receive"} 0
system_network_errors{device="en3",direction="transmit"} 0
system_network_errors{device="en4",direction="receive"} 0
system_network_errors{device="en4",direction="transmit"} 0
system_network_errors{device="en5",direction="receive"} 0
system_network_errors{device="en5",direction="transmit"} 1342
system_network_errors{device="gif0",direction="receive"} 0
system_network_errors{device="gif0",direction="transmit"} 0
system_network_errors{device="llw0",direction="receive"} 0
system_network_errors{device="llw0",direction="transmit"} 0
system_network_errors{device="lo0",direction="receive"} 0
system_network_errors{device="lo0",direction="transmit"} 0
system_network_errors{device="stf0",direction="receive"} 0
system_network_errors{device="stf0",direction="transmit"} 0
system_network_errors{device="utun0",direction="receive"} 0
system_network_errors{device="utun0",direction="transmit"} 0
system_network_errors{device="utun1",direction="receive"} 0
system_network_errors{device="utun1",direction="transmit"} 0
# HELP system_network_io The number of bytes transmitted and received.
# TYPE system_network_io counter
system_network_io{device="ap1",direction="receive"} 0
system_network_io{device="ap1",direction="transmit"} 0
system_network_io{device="awdl0",direction="receive"} 0
system_network_io{device="awdl0",direction="transmit"} 52438
system_network_io{device="bridg",direction="receive"} 0
system_network_io{device="bridg",direction="transmit"} 0
system_network_io{device="en0",direction="receive"} 8.68152775e+08
system_network_io{device="en0",direction="transmit"} 3.8290379e+08
system_network_io{device="en1",direction="receive"} 0
system_network_io{device="en1",direction="transmit"} 0
system_network_io{device="en2",direction="receive"} 0
system_network_io{device="en2",direction="transmit"} 0
system_network_io{device="en3",direction="receive"} 0
system_network_io{device="en3",direction="transmit"} 0
system_network_io{device="en4",direction="receive"} 0
system_network_io{device="en4",direction="transmit"} 0
system_network_io{device="en5",direction="receive"} 2.060673e+06
system_network_io{device="en5",direction="transmit"} 8.734926e+06
system_network_io{device="gif0",direction="receive"} 0
system_network_io{device="gif0",direction="transmit"} 0
system_network_io{device="llw0",direction="receive"} 0
system_network_io{device="llw0",direction="transmit"} 0
system_network_io{device="lo0",direction="receive"} 5.68669269e+08
system_network_io{device="lo0",direction="transmit"} 5.68669269e+08
system_network_io{device="stf0",direction="receive"} 0
system_network_io{device="stf0",direction="transmit"} 0
system_network_io{device="utun0",direction="receive"} 0
system_network_io{device="utun0",direction="transmit"} 1726
system_network_io{device="utun1",direction="receive"} 0
system_network_io{device="utun1",direction="transmit"} 1726
# HELP system_network_packets The number of packets transferred.
# TYPE system_network_packets counter
system_network_packets{device="ap1",direction="receive"} 0
system_network_packets{device="ap1",direction="transmit"} 0
system_network_packets{device="awdl0",direction="receive"} 0
system_network_packets{device="awdl0",direction="transmit"} 482
system_network_packets{device="bridg",direction="receive"} 0
system_network_packets{device="bridg",direction="transmit"} 0
system_network_packets{device="en0",direction="receive"} 1.10909e+06
system_network_packets{device="en0",direction="transmit"} 1.53465e+06
system_network_packets{device="en1",direction="receive"} 0
system_network_packets{device="en1",direction="transmit"} 0
system_network_packets{device="en2",direction="receive"} 0
system_network_packets{device="en2",direction="transmit"} 0
system_network_packets{device="en3",direction="receive"} 0
system_network_packets{device="en3",direction="transmit"} 0
system_network_packets{device="en4",direction="receive"} 0
system_network_packets{device="en4",direction="transmit"} 0
system_network_packets{device="en5",direction="receive"} 12718
system_network_packets{device="en5",direction="transmit"} 13205
system_network_packets{device="gif0",direction="receive"} 0
system_network_packets{device="gif0",direction="transmit"} 0
system_network_packets{device="llw0",direction="receive"} 0
system_network_packets{device="llw0",direction="transmit"} 0
system_network_packets{device="lo0",direction="receive"} 5.774042e+06
system_network_packets{device="lo0",direction="transmit"} 5.774042e+06
system_network_packets{device="stf0",direction="receive"} 0
system_network_packets{device="stf0",direction="transmit"} 0
system_network_packets{device="utun0",direction="receive"} 0
system_network_packets{device="utun0",direction="transmit"} 11
system_network_packets{device="utun1",direction="receive"} 0
system_network_packets{device="utun1",direction="transmit"} 11
# HELP system_paging_faults The number of page faults.
# TYPE system_paging_faults counter
system_paging_faults{type="major"} 0
system_paging_faults{type="minor"} 0
# HELP system_paging_operations The number of paging operations.
# TYPE system_paging_operations counter
system_paging_operations{direction="page_in",type="major"} 0
system_paging_operations{direction="page_in",type="minor"} 0
system_paging_operations{direction="page_out",type="major"} 0
system_paging_operations{direction="page_out",type="minor"} 0
# HELP system_paging_usage Swap (unix) or pagefile (windows) usage.
# TYPE system_paging_usage gauge
system_paging_usage{state="cached"} 0
system_paging_usage{state="free"} 0
system_paging_usage{state="used"} 0
# HELP system_processes_count Total number of processes in each state.
# TYPE system_processes_count gauge
system_processes_count{status="blocked"} 0
system_processes_count{status="running"} 2

[fage88]

Is there any update on this? any idea when this might get worked?

[pmeresanu85]

@fage88 Apologies for the delay in updating this ticket. Due to priority changes at the company level, we weren't able to come back with a reply in due time.

Having said that, we are currently working on scoping & designing how we can properly map metrics coming from Host Metric Receiver to host which will be visible in the Infrastructure part of the product, through the new host centric view as well as through the classic inventory (aka waffle map)

Our intent is to make this available at the host, but also at the host group level.

We are currently working on researching the best ways to translate metrics into ECS.

Expect an update on this topic post 8.5 release.

[ahmedmouradrwe]

@pmeresanu85 thanks for the reply, I would like to follow up on the progress of translating metrics into ECS, but I cannot find the issue https://github.com/elastic/observability-dev/issues/2293 , could you please confirm the issue number ? thanks

[wbarnha]

Has there been any progress made on this? It's something I definitely need because host metrics are quite tedious to organize.

[lewis-prior-thrivent]

Has any update on this?

[iamjosh007]

Any update on this?

[axw]

Apologies for the silence. Nothing much to update on yet, other than to say that we are working on various things that will make this possible. Eventually the Infra UI should be able to visualise these metrics, but it's TBD how that will work.

You may have seen recently that Elastic Common Schema and OpenTelemetry Semantic Conventions are converging -- so long term, things should be much simpler. Until that's finalised, some additional work is needed to either query both ECS/OTel, or to transform one to the other. Stay tuned, we'll update this when there's more to share.

[sardanam]

Do we have any update here ? we are adopting elastic as our observability solution. Lack of UIs around host matrices and k8s dashboards based on otel Schema is making visualization data very difficult. Till the time this is not fixed, can we have some custom dashboard template, json template to reuse and create custom dashboards ?

[Maxime-CLS]

Any update on this?

[mlunadia]

Hi folks, As many of you are probably aware Elastic has been a contributor of the OpenTelemetry project and since the donation of ECS to the OpenTelemetry project we have ramped up our efforts towards converging both standards and avoid these kind of issues. Given that convergence is a process that implies changes to both standards and will require time to agree, we are currently evaluating options to solve these challenging problems of UIs supporting 2 concurrent standards and the evolution of OTel semantic conventions (which are currently experimental). Feel free to DM me as I will be interested to understand a broader set of use cases and make sure we cover as much as we can with the work we are doing on this.