esc
esc copied to clipboard
pulumi
Support pinning reference to environments to specific versions.
Today, any reference to an environment implicitly references the latest edit to that environment. This has some benefits as it makes it easy to make a change one place and have it picked up elsewhere without forcing all consumers to also change, but also makes it harder to ensure that a given consumer can be in control of when it adopts new configuration from an environment.
We would like to add support for specifying versions as part of an environment reference - for example:
# Pulumi.staging.yaml
environment:
- aws-staging@v5
A number of design questions we need to address:
- Are versions incremented every time an environment is saved? Or is there a tagging mechanism (like Docker) where users can explicitly specify these versions/tags/labels?
- Can a version/tag/label change, or are these truly immutable?
- Is there an implicit
latestwhich is used by default? - Can these versioned references be used in all positions where an environment is referenced?
- Do we support git hash or tag?
- Likely too complicated but interesting: Do we support wildcards for semver? like import:
- aws-staging@5.*or package management- aws-staging<=6(admittedly seems like overkill)
If a version ends up being used to denote a version of the schema of the environment (the object structure), they should be mutable (like for secret rotation), and I would want to be pinned to a version.
If a version ends up being used to denote the values of the environment (thinking of credentials that may be rotated), then it would be fine with me if they were immutable, but I would generally want to be at latest.
My very lightly held opinion is that we should intend a version to mean a version of the schema (i.e. the "public interface"), not a version of the values. In either case, I think we should be explicit about how we intend a versioning feature to be used. This is a new tool, so I think prescriptive guidance will be helpful in getting users to understand the usefulness of the tool.
I would also recommend thinking through some common use cases for end users as we design this, which can then inform the eventual user-facing documentation for the feature.
I would prefer to implement a [previous, current, next] mechanism, to avoid race conditions.
Having three keys allows graceful rotation of the tickets. Services issue tickets with the “current” keys and accept tickets with the previous and next keys. The STEKs are stored inside the container, and the service reloads the STEKs whenever they change. - Building Facebook’s service encryption infrastructure
When changing credentials, there can be a lag between a value being created and accepted. Allowing clients to operate on the "current" key while the validator accepts each is a safer practice.
A Pulumi ESC resource for managing an ESC key would be perfect for implementing key rotation, JWKS, database credentials.
For my use case I'd want at minimum some kind of audit trail showing who changed something and what changed. This is why we use IaC in the first place - to have version controlled infrastructure and visibility into changes. Having the extra features mentioned above like pinning to a schema version would be a nice bonus.
We will be taking the following approach for our first cut:
We will provide a simple, Docker-style approach to versioning that lays the foundations we will need for a more sophisticated approach in the future. This approach involves two new concepts--revisions and tags--and an extension to the import syntax.
Revisions
A revision to and environment captures the changes made by a single edit to an environment. A new revision is created with each edit, even if the edit does not make any changes to the environment definition.
Revisions are identified by revision numbers. Revision numbers start from 1 and form a dense, monotonically-increasing sequence. That is, the first edit to an environment is revision 1, the second is revision 2, and so on.
Tags
A tag is a mutable reference to a particular revision. A tag may only be associated with a single revision at a time. There is one builtin tag, latest, that is created by ESC itself and is updatedto point to the current revision of an environment after each call to UpdateEnvironment. Following Docker, tags must be valid ASCII and can contain lowercase and uppercase letters, digits, underscores, periods, and hyphens, and must be no longer than 128 characters. We will additionally constrain tags such that they may not begin with a digit. This allows us to easily distinguish tags from revisions.
Only users with WRITE access to an environment will be allowed to create or update tags.
Import syntax
We will extend the import syntax in order to allow environments to track specific revisions or tags. Currently, an imported environment must be a legal environment name, which limits its contents to alphanumeric strings. We will extend this to allow a :[revision-or-tag] suffix. If revision-or-tag begins with a digit, we will treat it as a revision number, and the import will refer to the specific revision of the named environment. If revision-or-tag does not begin with a digit, we will treat itas a tag, and the import will refer to whatever revision is pointed to by the tag at the time the import is evaluated. If no revision-or-tag is present, the import will implicitly refer to thelatest tag.
This syntax and the existing constraints on environment names leave open the possibility of future syntax for refering to semvers should we choose to pursue that avenue (e.g. [email protected]).
Console UI/UX
We will extend the console UI for an environment to support the following operations:
- Viewing the revision history for an environment
- Viewing the tags for an environment
- Tagging revisions of an environment
We will make similar extensions to the esc and pulumi env CLIs.
This is done. Blog post is here: https://www.pulumi.com/blog/esc-versioning-launch/