omni icon indicating copy to clipboard operation
omni copied to clipboard

[bug] Pushing of machine logs seems expensive

Open rsmitty opened this issue 10 months ago • 2 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Current Behavior

For customers with lots of nodes, in monitoring we see the 10m timer where we push machine logs to the ceph volumes seems to heavily saturate the I/O.

Expected Behavior

This is maybe an exploration of how we might make this less heavy (if we even determine it's something we should be worried about).

Steps To Reproduce

N/A

What browsers are you seeing the problem on?

No response

Anything else?

Screenshot from 2024-04-08 11-05-59

^ An example, but I've seen quite a bit higher percentage than this. Saw over 300% last week.

rsmitty avatar Apr 08 '24 15:04 rsmitty