Move the reconciliation loop to a model where each run executes as a k8s Job

[metral]

Problem description

Currently, the reconciliation loop is shared in-process on the operator.

Ideally, a separate Job/Pod would be used instead per loop to avoid shared context issues across loops and runtimes. See the design doc.

[EronWright]

Using a Pod to execute pulumi up may unlock some interesting capabilities. Some example:

1. mounting of stack-specific volumes to act as an alternate source for the program sources, program data, or even backend state.
2. Stack-specific service accounts. This would allow us to harden the permissions granted to the operator itself.
3. Clean separation of log output between the operator and the program.

[viveklak]

Note that if we support https://github.com/pulumi/pulumi-kubernetes-operator/issues/122 we will have to ensure the secret injection mechanism can mount secrets into the job/pod encapsulating the stack.