Retry on 429-IdcsConversionError

[EladGabay]

Currently IDCS throttled errors are returned with IdcsConversionError and not with the underline TooManyRequests. That means, the SDK won't retry on these errors, but it should.

For example:

ErrorCode - 429 - IdcsConversionError ErrorMessage - GET request failed{"schemas":["urn:ietf:params:scim:api:messages:2.0:Error","urn:ietf:params:scim:api:oracle:idcs:extension:messages:Error"],"detail":"Too many GET requests received from Stripe idcs-XXXX initiated by client IP 'X0.X04.XX.X8' on endpoint 'admin/v1/Users'","status":"429","urn:ietf:params:scim:api:oracle:idcs:extension:messages:Error":{"messageId":"error.common.ratelimiting.stripe.toomanyrequests"}}

[EladGabay]

@swkrkris @ZiyaoQiao @waruwaruwaru

[jodoglevy]

@EladGabay IDCS is not currently exposed in the OCI SDKs. What SDK operation is giving you a IdcsConversionError?

[EladGabay]

@jodoglevy it's OCI IAM APIs like listApiKeys, etc . In tenancies with identity domain

[jodoglevy]

Let me talk to the Identity team. I think their service should still be returning the standard OCI error codes to you, even if they interface with IDCS behind the scenes. IdcsConversionError is a non-standard error, not one OCI documents to customers as something we'll return, and "IdcsConversionError" seems pretty generic and does not seem to indicate it is only returned for cases of a "TooManyRequests" throttling issue. We don't want the SDK to retry on IdcsConversionError for conversion error cases not related to throttling, for example.

[EladGabay]

I agree and already opened SR. In the meantime, I got many errors in production because no retry in this case, so I think it's better to have this support in the SDK, until they will fix it (might take some time..)

[EladGabay]

And note that it's 429 specifically.

[ZiyaoQiao]

Hi, this error code is not a standard OCI error code so SDK is not adding it at this moment. We'll check with Identity on this. Meanwhile, I'd recommend using a customized retry policy to handle this case. In SDK, this status code is defined for the default retry policy and you can customize it by creating your customizedShouldRetryOperation. Some code snippets you could refer:

var (
	yourCustomizedRetryStatusCodeMap = map[StatErrCode]bool{
		{409, "IncorrectState"}:  true,
		{429, "TooManyRequests"}: true,
		{429, "IdcsConversionError"}: true,
  
		{501, "MethodNotImplemented"}: false,
	}
)

func YourCustomizedShouldRetryOperation(r OCIOperationResponse) bool {
	if r.Error == nil && 199 < r.Response.HTTPResponse().StatusCode && r.Response.HTTPResponse().StatusCode < 300 {
		// success
		return false
	}
	return IsErrorRetryableByDefault(r.Error)
}


func YourCustomizedIsErrorRetryable(Error error) bool {
	if Error == nil {
		return false
	}

	if IsNetworkError(Error) {
		return true
	}

	if err, ok := IsServiceError(Error); ok {
		if shouldRetry, ok := yourCustomizedRetryStatusCodeMap[StatErrCode{err.GetHTTPStatusCode(), err.GetCode()}]; ok {
			return shouldRetry
		}

		return 500 <= err.GetHTTPStatusCode() && err.GetHTTPStatusCode() < 505
	}

	return false
}

and you could config the retryPolicy this way:

	retryPolicy := common.DefaultRetryPolicy()
        // If you want to configure and use it for Eventually Consistent Retry
	retryPolicy.ShouldRetryOperation = YourCustomizedShouldRetryOperation
        // If you want to configure and use it for Non Eventually Consistent Retry / common case retry
	retryPolicy.NonEventuallyConsistentPolicy.ShouldRetryOperation = YourCustomizedShouldRetryOperation