vector icon indicating copy to clipboard operation
vector copied to clipboard
verified publisher icon vectordotdev

host_metrics - linux - Invalid netlink buffer length

Open zapdos26 opened this issue 10 months ago • 6 comments

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Getting the following error for the new TCP metric:   2025-02-20T22:54:35.389172Z ERROR source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector::internal_events::host_metrics: Failed to load tcp connection info. error=Could not parse netlink response: Decode error occurred: invalid netlink buffer: length field says 524293 the buffer is 4008 bytes long error_type="reader_failed" stage="receiving" internal_log_rate_limit=true

Configuration

sources:
  host_metrics:
    type: host_metrics

sinks:
  console:
    type: file
    path: host_metrics-text.log
    inputs:
      - host_metrics
    encoding:
      codec: json
~

Version

vector 0.45.0-custom-ae84bcd9d (x86_64-unknown-linux-gnu debug=full)

Debug Output

2025-02-20T23:05:55.675548Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_common::internal_event::bytes_received: Bytes received. byte_size=0 protocol=none
2025-02-20T23:05:55.676023Z DEBUG sink{component_kind="sink" component_id=console component_type=file}: vector::utilization: utilization=0.0024510706853726866
2025-02-20T23:05:56.183700Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}:grow_pool{queue_len=1 idle_count=0 thread_count=0}: blocking: spawning a new thread to handle blocking tasks
2025-02-20T23:05:56.185299Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.185891Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.186367Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.187020Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.187666Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.194702Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.196196Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.235156Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.236012Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.240930Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.241963Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.243467Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.248093Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.254284Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.254950Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.256683Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.257656Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.259002Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.261414Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.290018Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.290613Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.292088Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.292833Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.294907Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.295929Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.297755Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.298492Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.300316Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.301127Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.301940Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.302261Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.303175Z TRACE blocking::main_loop: blocking: notified
2025-02-20T23:05:56.306684Z TRACE blocking::main_loop: blocking: going to sleep timeout=500ms
2025-02-20T23:05:56.308283Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: mio::poll: registering event source with poller: token=Token(140591263223168), interests=READABLE | WRITABLE    
2025-02-20T23:05:56.311475Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: mio::poll: deregistering event source from poller    
2025-02-20T23:05:56.312198Z ERROR source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector::internal_events::host_metrics: Failed to load tcp connection info. error=Could not parse netlink response: Decode error occurred: invalid netlink buffer: length field says 524293 the buffer is 3304 bytes long error_type="reader_failed" stage="receiving" internal_log_rate_limit=true
2025-02-20T23:05:56.313950Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_common::internal_event::events_received: Events received. count=2100 byte_size=1705753
2025-02-20T23:05:56.316057Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_buffers::topology::channel::limited_queue: Sent item.
2025-02-20T23:05:56.316248Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_common::internal_event::events_sent: Events sent. count=1000 byte_size=756179 output=_default
2025-02-20T23:05:56.317948Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_buffers::topology::channel::limited_queue: Sent item.
2025-02-20T23:05:56.318124Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_common::internal_event::events_sent: Events sent. count=1000 byte_size=875676 output=_default
2025-02-20T23:05:56.318319Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_buffers::topology::channel::limited_queue: Sent item.
2025-02-20T23:05:56.318334Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_common::internal_event::events_sent: Events sent. count=100 byte_size=73900 output=_default
2025-02-20T23:05:56.319710Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_buffers::topology::channel::limited_queue: Sent item.
2025-02-20T23:05:56.319950Z TRACE source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector_core::fanout: Sent item to fanout.
2025-02-20T23:05:56.322177Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector_common::internal_event::events_received: Events received. count=1000 byte_size=756179
2025-02-20T23:05:56.322355Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector::sinks::file: Computed next deadline. next_deadline=Instant { tv_sec: 189110, tv_nsec: 537303951 } path=b"host_metrics-text.log"
2025-02-20T23:05:56.322477Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector::sinks::file: Working with an already opened file. path=b"host_metrics-text.log"
2025-02-20T23:05:56.322493Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector::sinks::file: Writing an event to file. path=b"host_metrics-text.log"
2025-02-20T23:05:56.323026Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=566
2025-02-20T23:05:56.323092Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector::internal_events::file: Bytes sent. byte_size=209 protocol="file" file=host_metrics-text.log
2025-02-20T23:05:56.323528Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector::sinks::file: Computed next deadline. next_deadline=Instant { tv_sec: 189110, tv_nsec: 538459439 } path=b"host_metrics-text.log"
2025-02-20T23:05:56.323629Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector::sinks::file: Working with an already opened file. path=b"host_metrics-text.log"
2025-02-20T23:05:56.323645Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector::sinks::file: Writing an event to file. path=b"host_metrics-text.log"
2025-02-20T23:05:56.323940Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector_common::internal_event::events_sent: Events sent. count=1 byte_size=566
2025-02-20T23:05:56.323987Z TRACE sink{component_kind="sink" component_id=console component_type=file}: vector::internal_events::file: Bytes sent. byte_size=209 protocol="file" file=host_metrics-text.log

Example Data

No response

Additional Context

No response

References

No response

zapdos26 avatar Feb 20 '25 23:02 zapdos26

Hi @zapdos26, what platform/kernel are you running this on? I wonder if there's an incompatibility.

pront avatar Feb 25 '25 21:02 pront

I am getting this same error with a very similar config. Here is the output of uname -a

Linux desktop 6.9.3-76060903-generic #202405300957~1736980680~22.04~44ea8a9~dev-Ubuntu SMP PREEMPT_DY x86_64 x86_64 x86_64 GNU/Linux

I installed vector via the .deb package instructions from your website. Vector version:

% vector --version vector 0.45.0 (x86_64-unknown-linux-gnu 063cabb 2025-02-24 14:52:02.810034614)

configuration file 
[sources.system_metrics]
type = "host_metrics"

[transforms.metrics_to_logs]
type = "metric_to_log"
inputs = ["system_metrics"]

[transforms.format_logs]
type = "remap"
inputs = ["metrics_to_logs"]
source = '''
  .timestamp = now()
  .host, _ = get_hostname()
'''

[sinks.csv_output]
type = "file"
inputs = ["format_logs"]
path = "/tmp/stats/stats.json"
compression = "none"
encoding.codec = "json"

[sinks.csv_output.buffer]
type = "disk"
max_size = 268435488

dontlaugh avatar Mar 16 '25 22:03 dontlaugh

Hmm, I can see TCP metrics on:

$ uname -a
Linux i-01ce621161b2afdd2 6.8.0-1024-aws #26-Ubuntu SMP Tue Feb 18 17:22:37 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

I suspect that the underlying dependencies do not support unix properly.

pront avatar Mar 17 '25 17:03 pront

Same issue here: The config file just has host_metrics source and not much else

// uname -a
Linux staging-1 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64 GNU/Linux

// root@staging-1:~# vector --version
// vector 0.45.0 (x86_64-unknown-linux-gnu 063cabb 2025-02-24 14:52:02.810034614)

afjoseph avatar Mar 23 '25 12:03 afjoseph

+1

Using 0.44.0-1 does not experience this issue. This problem seems to be happening for ver 0.45.0-1 recently.

techministrator avatar Apr 09 '25 09:04 techministrator

i cant reproduce this on my machine with vector 0.47.0

$ uname -a
Linux smth 6.12.15-orbstack-00304-gd0ddcf70447d #60 SMP Tue Feb 18 19:55:47 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

aryan9600 avatar May 30 '25 08:05 aryan9600

Observed similar just now. The host_metrics source has no other config settings.

Jul 27 04:35:19 stargazer vector[41971]: 2025-07-27T03:35:19.686254Z ERROR source{component_kind="source" component_id=host component_type=host_metrics}: vector::internal_events::host_metrics: Failed to load tcp connection info. error=Invalid message length: 524293 error_type="reader_failed" stage="receiving"

root@stargazer:~# vector --version
vector 0.48.0 (aarch64-unknown-linux-gnu a67e4e2 2025-06-30 18:25:45.272082383)

root@stargazer:~# uname -a
Linux stargazer 6.12.34+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.34-1+rpt1~bookworm (2025-06-26) aarch64 GNU/Linux

Cellivar avatar Jul 27 '25 03:07 Cellivar