Do the commit check before setting up workdir

[JonCholas]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Today, pulumi operator checks if the commit of a stack has changed after creating the working directory. Creating the working directory involves installing all the dependencies just to check a commit and stop there because the commit hasn't changed.

This is a waste of resources and time as when we want to update the stack, we will not see the pulumi operator doing anything until it has finished preparing the working directory for the previous execution.

In the logs(at the bottom), you can see how it

• Reconcilling Stack 2023-05-29T03:54:10.676Z
• Commit check 2023-05-29T03:55:00.273Z

It takes a minute to check a commit, and is doing it every minute so there is a high chance that everytime we change a config/commit we have to wait up to a minute for the previous run of pulumi check to finish.

Currently you are calling:

revision, err := revisionAtWorkingDir(w.WorkDir())

And then not returning it until the workdir is setup, and therefore, dependencies installed

return revision, sess.setupWorkspace(ctx, w)

Can we move the creation of workdir outside and check the revision earlier?

2023-05-29T03:54:10.676Z - Reconciling Stack - -  -
2023-05-29T03:54:10.696Z - Setting up pulumi workdir for stack - -  -
2023-05-29T03:54:12.107Z - Using local stack - -  -
2023-05-29T03:54:15.483Z - Setting autostack - -  -
2023-05-29T03:54:18.693Z - Initial autostack config - -  -
2023-05-29T03:54:18.694Z - stackConfig loaded - -  -
2023-05-29T03:54:21.984Z - Updated stack config - -  -
2023-05-29T03:54:21.984Z - InstallProjectDependencies - -  -
2023-05-29T03:54:25.880Z - Pip Install - Requirement already satisfied: pip in /tmp/pulumi-working/beta-neara-f515a5a8/deployment-beta-beta/pulumi/venv/lib/python3.9/site-packages (22.0.4)-  -
2023-05-29T03:54:26.012Z - Pip Install - Collecting pip-  -
2023-05-29T03:54:26.021Z - Pip Install -   Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)-  -
2023-05-29T03:54:26.032Z - Pip Install - Requirement already satisfied: setuptools in /tmp/pulumi-working/beta-neara-f515a5a8/deployment-beta-beta/pulumi/venv/lib/python3.9/site-packages (58.1.0)-  -
2023-05-29T03:54:26.349Z - Pip Install - Collecting setuptools-  -
2023-05-29T03:54:26.354Z - Pip Install -   Using cached setuptools-67.8.0-py3-none-any.whl (1.1 MB)-  -
2023-05-29T03:54:26.411Z - Pip Install - Collecting wheel-  -
2023-05-29T03:54:26.413Z - Pip Install -   Using cached wheel-0.40.0-py3-none-any.whl (64 kB)-  -
2023-05-29T03:54:26.520Z - Pip Install - Installing collected packages: wheel, setuptools, pip-  -

....

2023-05-29T03:54:32.967Z - Pip Install - Collecting certifi>=2017.4.17 (from requests<3.0,>=2.21->pulumi-kubernetes<4.0.0,>=3.0.0->pulumi-neara==0.1.0->-r requirements.txt (line 1))-  -
2023-05-29T03:54:32.970Z - Pip Install -   Using cached certifi-2023.5.7-py3-none-any.whl (156 kB)-  -
2023-05-29T03:54:33.371Z - Pip Install - Installing collected packages: types-PyYAML, arpeggio, urllib3, typing-extensions, tomli, six, semver, pyyaml, pyflakes, pycodestyle, protobuf, platformdirs, pathspec, packaging, mypy-extensions, mccabe, isort, idna, grpcio, dill, click, charset-normalizer, certifi, attrs, requests, pulumi, parver, mypy, flake8, black, pulumi-random, pulumi-kubernetes, pulumi-gcp, pulumi-aws, pulumi-eks, pulumi-neara-  -
2023-05-29T03:54:59.447Z - Pip Install -   Running setup.py develop for pulumi-neara-  -
2023-05-29T03:54:59.874Z - Pip Install - Successfully installed arpeggio-2.0.0 attrs-23.1.0 black-23.1.0 certifi-2023.5.7 charset-normalizer-3.1.0 click-8.1.3 dill-0.3.6 flake8-6.0.0 grpcio-1.51.3 idna-3.4 isort-5.12.0 mccabe-0.7.0 mypy-1.2.0 mypy-extensions-1.0.0 packaging-23.1 parver-0.4 pathspec-0.11.1 platformdirs-3.5.1 protobuf-4.23.2 pulumi-3.68.0 pulumi-aws-5.41.0 pulumi-eks-1.0.2 pulumi-gcp-6.56.0 pulumi-kubernetes-3.28.1 pulumi-neara-0.1.0 pulumi-random-4.13.2 pycodestyle-2.10.0 pyflakes-3.0.1 pyyaml-6.0 requests-2.31.0 semver-2.13.0 six-1.16.0 tomli-2.0.1 types-PyYAML-6.0.0 typing-extensions-4.6.2 urllib3-2.0.2-  -
2023-05-29T03:55:00.273Z - Checking current HEAD commit hash - -  -
2023-05-29T03:55:00.273Z - Commit hash unchanged. Will poll again. - -  -

Affected area/feature

[rquitales]

Hi @JonCholas, thank you for reporting this enhancement. Your suggestion to reduce unnecessary computations and network calls sounds like a reasonable change that could greatly benefit the project. We appreciate your input and will add this enhancement to our backlog for further evaluation.

If you're interested, we would also welcome any contributions you may have to speed up the implementation of this enhancement. We value collaboration and would be glad to work with you in landing a pull request that addresses this feature. Please feel free to reach out if you have any questions or need assistance along the way.

[diablo39]

Idea looks fantastic! When we can expect implementation?

[EronWright]

Good news everyone, we just released a preview of Pulumi Kubernetes Operator v2. This new release has a whole-new architecture that uses pods as the execution environment. All checks are performed before the pod is provisioned, and the pod is retained across deployment ops (unless and until the commit has changed, at which time the pod is replaced). Installation happens once when the pod is initialized.

Please read the announcement blog post for more information: https://www.pulumi.com/blog/pulumi-kubernetes-operator-2-0/

Would love to hear your feedback! Feel free to engage with us on the #kubernetes channel of the Pulumi Slack workspace. cc @JonCholas @davide-romanini-milkman @kerruba-milkman