emissary icon indicating copy to clipboard operation
emissary copied to clipboard

Ambassador needs to ignore no-op "changes" to Kubernetes resources

Open kflynn opened this issue 4 years ago • 8 comments

A user recently ran into a situation where his Kubernetes Ingress resources were changing every few seconds, triggering an Ambassador reconfiguration even though the Ambassador installation wasn't using Ingress resources for its configuration, and even though (as I understand it) the changes to the resources weren't anything that Ambassador should've needed to care about even if Ambassador were supposed to be paying attention to Ingress resources.

Ambassador should filter out immaterial changes in watt, so that the more expensive reconfiguration doesn't need to happen at all.

kflynn avatar May 19 '20 14:05 kflynn

Hullo, just wanted to :+1: this issue - we find that when performing a deployment in kubernetes the number of resource changes seems to occasionally trigger our ambassador deployments to significantly increase their CPU utilisation.

This is particularly bad for internal clusters where your CI pipeline may be running in the same cluster as your ambassador deployments: the CPU utilisation of ambassador can degrade the speed of your pipeline, causing it to take longer to update kubernetes resources, causing ambassador to be confused & high-CPU for longer, causing the pipeline to take longer, etc.

Do you have a sense of the time scales for this issue, and, is there anything I could do to assist? Might be able to provide information, or be able to spend time investigating the issue to create a PR (though I've never worked on this code base, so mileage may vary :smile:)

omaskery avatar Jun 02 '20 14:06 omaskery

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Aug 01 '20 14:08 stale[bot]

not stale

jwm avatar Aug 06 '20 01:08 jwm

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Oct 05 '20 01:10 stale[bot]

not sure if the ~recent fast reconfig support takes care of this/makes this a non-issue.

jwm avatar Oct 05 '20 16:10 jwm

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Dec 04 '20 21:12 stale[bot]

This still happens on legacy mode, which we are currently stuck on due to issues with Consul 1.10.

lawliet89 avatar Sep 29 '21 09:09 lawliet89

not stale

EvertonSA avatar Sep 02 '22 21:09 EvertonSA