Bug: Inventory updates should tolerate drift (and overwrite it)

[karlkfi]

Right now, inventory updates may return a conflict error from Kubernetes. The inventory client should detect this (apierrors.IsConflict(err)) and retry with a new Get (to update the ResourceVersion) + Update.

Example retry code:

type retriable func(ctx context.Context) (retry bool, err error)

func retryWithBackoff(ctx context.Context, timeout time.Duration, fn retriable) error {
	var err error
	var retry bool
	ctx, cancel := context.WithTimeout(ctx, timeout)
	defer cancel()
	delay := 1 + time.Second
	for {
		// attempt to update status
		retry, err = fn(ctx)
		if !retry {
			return err
		}

		// wait until delay or timeout
		timer := time.NewTimer(delay)
		select {
		case <-ctx.Done():
			timer.Stop()
			return fmt.Errorf("timed out after retrying for %v: %w", timeout, err)
		case <-timer.C:
			// continue
		}
		// retry backoff
		delay = delay * 2
	}
}

example usage:

	// attempt to update status until timeout
	ctx := context.TODO()
	timeout := 1 * time.Minute
	return retryWithBackoff(ctx, timeout, func(ctx context.Context) (retry bool, err error) {
		// Get the object to get the latest ResourceVersion.
		latestObj, err := resource.Get(ctx, obj.GetName(), metav1.GetOptions{TypeMeta: meta})
		if err != nil {
			return false, fmt.Errorf("failed to get inventory status from cluster: %w", err)
		}
		// Ignore any status changes made remotely.
		// This update will replace them.
		obj.SetResourceVersion(latestObj.GetResourceVersion())

		_, err = resource.UpdateStatus(ctx, obj, metav1.UpdateOptions{TypeMeta: meta})
		if err != nil {
			// retry if conflict
			return apierrors.IsConflict(err), fmt.Errorf("failed to write updated inventory status to cluster: %w", err)
		}
		return false, nil
	})

Another option is to use https://github.com/flowchartsman/retry which is nice and generic. gcloud and client-go also have retry libs.

[karlkfi]

The main client causing drift right now is the Config Sync resource-group-controller, which updates the ResourceGroup (inventory) status.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[karlkfi]

/remove-lifecycle rotten /lifecycle frozen