vector icon indicating copy to clipboard operation
vector copied to clipboard

Published 20 hours ago • verified publisher icon vectordotdev

Vector high memory utilization

Open Karjan1 opened this issue 8 months ago • 4 comments

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hi, we have one of the environments behaving strange. After each deployment on that env, memory usage of Vector spikes to dangerous values. It's a preprod environment with the same configuration as prod environments, but much lower traffic per Vector container. Some numbers:

Common for all envs:

  • We're using pretty large containers with 13vCPUs and 26GB of memory
  • We have autoscaling set 20% memory and 65% CPU utilization
  • We have a lot of transformations, this impacts Vector startup time that is ~9minutes.
  • For buffer_full on all sinks, we're dropping newest.

Prod:

  • CPU utilization is cycling at around 50-90% (occasional spikes to 100%)
  • Memory is usually ~3-4GB (occasionally spikes to 6GB)
  • With those numbers Vector processes data reliably, without drops.

Preprod:

  • Single preprod container processes around 15% of traffic going through prod one
  • CPU utilization is ~20-25%. During deploys spikes to 40-50%
  • Memory is usually ~1GB. During deploys spikes to 11-13GB (50-60%) (rarely reaches 13GB, it can cause container to crash)

Below is an image of memory utilization of one container after deploy vector_preprod

My guess about what could be possibly happening is that Vector struggles with quick scaling of a single transformation. In our case vector has a lot of transformations that are specific to a single service. And preprod takes traffic from a small set of services. When looking at utilization of transformations after deploy, we see spikes but it looks relatively healthy and nothing else is different on that env, so I'm probably wrong.

Utilization of each transformation vector_transformation_utilization

Happy to take any suggestions what might be the issue hee

Configuration

No response

Version

vector 0.38.0 (aarch64-unknown-linux-musl ea0ec6f 2024-05-07 14:34:39.794027186)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

Karjan1 avatar Jun 11 '24 10:06 Karjan1