Use ephemeral containers for upgrades

[johscheuer]

What would you like to be added/changed?

With the new unified image approach we should consider/explore to make use of ephemeral containers during upgrades to copy the new binaries inside the main container. The benefit of this approach is that a cluster doesn't "waste" resources for running a sidecar that actually doesn't do anything and those resources are only required when the binary is copied. There will be some interesting cases that we have to solve like the new binary was copied to the main container and the Pod get's deleted recreated, now we have to initiate the copy again.

Currently the feature is beta for Kubernetes 1.23+ so we could explore it for newer versions and add a feature gate to enable/disable this mode. It probably makes sense to write down a design doc on how this will work in detail.