Support Retry() while using response iterator

[ampkeegan]

Per recommendation of support case GCP #51120117 I'm opening a ticket here. Not sure if this is a bug report, feature request, or I'm just doing something wrong, but it appears the basic Retry() functionality for google-cloud-channel (and maybe others) only works if the limit is reached when the initial request is made. If a limit is reached while paging through the response iterator, Retry() is not implemented.

Example code:

_RETRYABLE_REASONS = frozenset(
[StatusCode.RESOURCE_EXHAUSTED, ]
)
 
_UNSTRUCTURED_RETRYABLE_TYPES = (
	   ConnectionError,
	   exceptions.Aborted,
	   exceptions.DeadlineExceeded,
	   exceptions.InternalServerError,
	   exceptions.ResourceExhausted,
	   exceptions.ServiceUnavailable,
	   exceptions.Unknown,
	   exceptions.Cancelled,
	   exceptions.TooManyRequests
)

def _should_retry(exc):
	if not hasattr(exc, "errors") or len(exc.errors) == 0:
		# Check for unstructured error returns, e.g. from GFE
		return isinstance(exc, _UNSTRUCTURED_RETRYABLE_TYPES)
	reason = exc.errors[0].code()
	return reason in _RETRYABLE_REASONS
 
custom_retry = Retry(
			   initial=1, # seconds (default: 0.1)
			   maximum=90.0, # seconds (default: 60.0)
			   multiplier=5, # default: 1.3
			   deadline=300.0, # seconds (default: 60.0)
			   predicate=_should_retry,
	   )

request = channel_v1.ListCustomersRequest(parent="accounts/" + channelAccount, page_size=50)
response = client_v1.list_customers(request, retry=custom_retry)

c = 0
for x in response.pages:
	print(c)
	c += 1

Running three simultaneous executions of this script will result in a RESOURCE_EXHAUSTED error. Immediately running a single instance of the script (while still within the window to be rate limited) will properly implement the Retry() with exponential backoff until the limit is removed. Once it can progress, it will fail again shortly while paginating through response.pages with another RESOURCE_EXHAUSTED.