datadog-operator icon indicating copy to clipboard operation
datadog-operator copied to clipboard

Published 20 hours ago • verified publisher icon DataDog

Update SLO reconcialition loop to avoid creating duplicate SLOs

Open philippeVV opened this issue 9 months ago • 6 comments

What does this PR do?

We observed that the SLO controller creates duplicate SLO when running with more than 1 replica. (issue 1062)

The PR fixes the issue by adding a new SyncStatus state. The "creating SLO" syncStatus is used to ensure that only one reconcile loop can be creating an SLO at the time.

Motivation

(issue 1062)

Additional Notes

N/A

Minimum Agent Versions

N/A

Describe your test plan

When running the latest version of the operator with more than 1 replica, we observed that a DatadogSLO would create 2 SLOs in Datadog. When building the PR, you will observe that only one SLO is created when applying a DatadogSLO.

Checklist

  • [ ] PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
  • [ ] PR has a milestone or the qa/skip-qa label

philippeVV avatar May 10 '24 16:05 philippeVV

Hey @khewonc @celenechang,

Can I have some help merging this PR and adding the required labels?

First-time contributor, please let me know if I need to change anything

philippeVV avatar May 10 '24 18:05 philippeVV

@philippeVV thank you very much for your contribution. I happened to be working on this issue as well; would you be willing to test an alternative solution if/when I push my branch? Or perhaps you can share the SLO manifest you used? Thank you.

celenechang avatar May 14 '24 12:05 celenechang

@celenechang Yes, I'll be interested to see/test your implementation. In my test, I used the SLO defined in the config/samples/datadoghq_v1alpha1_datadogslo.yaml.

philippeVV avatar May 14 '24 13:05 philippeVV

@philippeVV PR opened at #1185. Thank you

celenechang avatar May 15 '24 12:05 celenechang

Hey @celenechang,

You mentioned on #1185 that you had some reservations regarding the changes in this PR. Can you please elaborate? While we didn't fully understand the issue and it's not reliably reproducible, updating the syncStatus before creating the SLO eliminates the possibility of duplicate SLO.

Thank you

philippeVV avatar May 24 '24 17:05 philippeVV

@philippeVV, first thank you for your contribution and for helping with testing the other PR.

Our reservations are mainly around two points:

  • From our investigation root cause of duplication is concurrent reconciliation (and SLO API not being idempotent at least with current implementation). This PR doesn't address concurrent reconciliation and I believe creating duplicated SLOs with this approach is still matter of timing.
  • Proposed implementation may also cause reconciler to get stuck and never create an SLO. In order this change to work as intended below setting status and creating SLO should both succeed in the same reconcile. If first block succeeds and second fails, we end up with status set to DatadogSLOSyncCreating and never attempt to create the SLO again exiting here. Given r.checkRequiredTags() and r.create() make multiple API calls to API server and Datadog, this is quite possible.

Sorry, we didn't provide these details earlier. Let me know if you have more questions.

levan-m avatar May 24 '24 18:05 levan-m

Thanks for the feedback @levan-m, it's much appreciated.

I believe I understand the issues and will try to address them.

philippeVV avatar May 27 '24 22:05 philippeVV

@philippeVV did you have a chance to test the most recent Operator release? Any reason to keep this PR open?

levan-m avatar Jun 28 '24 19:06 levan-m

Hi @levan-m, I didn't get a chance to revisit the issue since then. I'll close the PR until I get time to test it again.

philippeVV avatar Jul 02 '24 15:07 philippeVV