Context Handling

[kensipe]

After fixing #202 with a solution of #207 ...

The fix in 207 is worth keeping as we just don't know what the passed in fn will do and it is better to handle a potential panic in Retry. However... the reason for the panics seen in the wild, is because; 1. there is a shared reference across go routines (which is the kube client) and 2. the testsuite is shutting down (clearing the client). What would be MUCH better is for a shared context to be cancelled. Context is already passed through to Retry, however Context is not as mature as we need (naively created as needed to satisfy the API).

What would you like to be added:

What we need:

1. create a kube client context (which is associated with kube client)
2. pass that context for all uses of kube client
3. cancel context during shutdown, but prior to nullifying client
4. make sure that Retry handles cancel correctly

or (after more thinking)

1. have state in the RetryClient which cancels it. The issue to be solved is... "who" manages and maintains the Context. Should it be passed by calling clients (which may want to control the cancelling among other things... or is it maintained by the retryclient which likewise needs to cancel things.

The issue will be if the RetryClient is made nil, then the state associated is lost. So... we either need defensive logic to make sure that is controlled... or we need to marry context of control with retryclient and calling client.

Those are my current thoughts..