Write RFC around performance/caching improvements

[bpmct]

We want to re-architect envbuilder in a way where it can run inside an existing devcontainer image (registry.company.com/envbuilder-images/python-project:latest) and pick up and rebuild any stale layers. This also includes a Terraform data source where it can check if there is an existing registry image for a repo/branch/Dockerfile or if the bare envbuilder image needs to be started. Envbuilder will also push the newly-built image

[mafredri]

Status update: We've written an RFC (Notion: Improving Workspace Startup Performance with Envbuilder), but we're still investigating the best path forward.

For now, we've decided to create two PoCs to validate our approach:

• [x] https://github.com/coder/envbuilder/issues/185
• [x] https://github.com/coder/envbuilder/issues/186

If we can prove these two methods work, we will be able to implement our ideal solution, and have options for further improvement.

(Moving this issue to blocked until validation is complete.)

[mafredri]

Both https://github.com/coder/envbuilder/issues/185 and https://github.com/coder/envbuilder/issues/186 have now been investigated and the approach for utilizing pre-built images has been validated. (See conclusions in aforementioned issues.)

See #213 for a reference implementation combining these PoCs.

The updated RFC is still pending approval, but in broad strokes we're looking to implement the following key components:

1. New subcommands (or implemented as options via environment variables)
   • envbuilder get-cached-image
   • envbuilder build
2. Embed the envbuilder binary in images
3. Sanitize the final container image for envbuilder
4. Two modes of operation
   1. Repo - use the repo/commit/hash as source of truth.
   2. Filesystem (current) - use the local files as source of truth.
5. Implement terraform provider for envbuilder
6. Future enhancements
   • Feed parsed devcontainer.json back into Terraform
   • Push Kaniko build cache layers as fully-fledged container images
   • Light-weight checkout for Repo-mode

(The details for each component have been omitted for brevity and pending approval.)

[mtojek]

Thanks for the update, Mathias. I will make sure to include the relevant issues in the current sprint.