Add a helper to run an arbitrary function with backoff

[easwars]

We have multiple places where we do a Thing and conditionally backoff before retrying the Thing. In many of these situations, the Thing decides if backoff state should be reset before the next retry.

Here is an example: https://github.com/grpc/grpc-go/blob/09fc1a349826f04420292e0915fd11a5ce3dc347/orca/producer.go#L147-L164

There are a few more like this being added to the new xdsClient transport.

We need to identify other locations as well.

[easwars]

Blocked on the xDS client API changes.

[dfawley]

Why is this blocked on that? I was expecting this to be an xDS agnostic API, since it also applies to things that are not xDS?

[easwars]

The changes required for this issue are not xDS specific. But given that currently we have 3 known usages of this pattern and two of them are in the pending xDS client API PR, I thought it would be better to get that PR in, and make this change, instead of getting lost in merge conflicts.