[kube-runtime] Lease Guard Implementation

[xrl]

Contributes an implementation towards https://github.com/kube-rs/kube-rs/issues/485

Motivation

Simply providing Object API support for Lease is not sufficient, we can provide a general Lease handle that periodically renews a Lease while the guard is alive.

TODO:

• [ ] Test cases which work under CI (could use guidance on this)
• [ ] Remove unwrap from backgrounded Lease renewal tokio task (how do we report errors in this situation?)
• [ ] Do we like the API? It's been sufficient for my needs to but we could kick it up a notch on the Rustiness.
• [ ] Bug with serde_json::json!() on the Patch calls. See line 129. I solved this in my other implementation by wrapping in a serde_yaml::to_string. Api(ErrorResponse { status: "Failure", message: "error decoding YAML: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal array into Go value of type map[string]interface {}", reason: "BadRequest", code: 400 })
• [ ] Experiment with making renewal task optional, require the user to spawn the task themselves or spawn their own renewal loop. I want to keep the Guard (drop the guard, drop the renewal task, zero out the lease) behavior.

Solution

1. Pull in the kube runtime
2. Check if your Lease is available
3. If available make a blocking call to acquire it (which will fail under losing contentious compare swap)
4. Perform your work while holding the Lease
5. Drop the Lease a channel signals an exit condition to the renewal task. Upon exiting the renewal loop the Lease is marked as available by nullifying its properties.

[nightkr]

Remove unwrap from backgrounded Lease renewal tokio task (how do we report errors in this situation?)

Hm, we also need some way to notify the user that the lease was lost (either because it expired, or because someone else "stole" it without waiting for your lease to expire, which might be a valid administrator action). I think that means that we might want to have a synchronous Lease::renew() method instead, which can notify about both of those conditions.

That kind of sucks ergonomically, but I think we could get around that for the common use case by defining something like (untested pseudocode warning!):

/// Waits for a lease to be acquired, and then runs the `Future` `f`
/// Cancels immediately and returns an error if the lease could not be renewed
pub async fn run_with_lease<F: Future>(f: F, lease: Lease) -> Result<F::Output> {
    lease.acquire().await?;
    let mut renew_interval = interval(renewal_period);
    pin_mut!(f);
    loop {
        select! {
            out = f => {
                lease.release().await?;
                return Ok(out);
            }
            _ = renew_interval.tick() => lease.renew().await?,
        }
    }
}

[clux]

Naming-wise, it's also possible we should call the worker here the Elector, and the params ElectorParams to avoid confusion with the existing Lease object.

[thedodd]

Hey folks, feel free to take a look at https://github.com/kube-rs/kube/pull/1056 as well. Stable, solid API, all parameterized, and matches the Golang core controller leader elector pattern.

[clux]

Closing this, this PR is not worked on. See #485 for more info later.