Explore self-describing config

[joeduffy]

At the moment, which config variables are required by a Pulumi program are completely opaque to the Pulumi runtime, CLI, and service. As a result, you need to run the program to figure out which config hasn't been set ... one config variable at a time. This yields a very poor experience, especially compared to competitive technologies. Ideally a program or library could advertise its config.

Here is one possible idea. There are many other avenues we could pursue, of course:

First, we introduce the notion of a "config module." Each Pulumi program can define one. There would be a default convention for each language host, so config.js in JavaScript and config.py in Python. The requirement for these modules is that they do not have side-effect, and especially that they do not allocate resources (something we can enforce); do very little other than read and prepare config.

This config module would contain statements very much like what we do today

import { Config } from "@pulumi/pulumi";
const config = new Config("program");
// region controls what AWS region to deploy into.
export let region: string = config.require("region");
// If createDomain is true, we will create a DNS entry.
export let createDomain: boolean = config.getBoolean("createDomain") || false;
// ...

except that each config variable may also include

• a description.
• a default (rather than ad-hoc code per the above).

(Note that, based on whether get or require is called, we already know if something is optional; and we can use the Boolean, etc. suffix to know what its expected type is.)

So, for instance:

import { Config } from "@pulumi/pulumi";
const config = new Config("program");
export let region: string = config.require("region", "the region to deploy into");
export let createDomain: boolean = config.getBoolean(
    "createDomain", false, "if true, create a DNS entry");
// ...

The CLI would know to execute this module if it wants to display config variables for the current program. This would give us the ability to display not only the currently set variables, but also those that need to be set. It would give us the ability to check all variables up front, to ensure they are set, before even running the program. And it would also even give us an interactive way of setting config:

$ pulumi config set
Enter "region" [required; the region to deploy into]: us-west-2
Enter "createDomain" [optional, boolean; if true, create a DNS entry]: 

Note that figuring out how to execute the transitive closure of modules might be tricky, except that I will note that we now have logic to find the transitive closure as part of computing required plugins.

If we go down a path like this, we could even consider changing our config APIs. They are already a bit ... strange. For example, we could make an API more like optimist (https://github.com/substack/node-optimist), that used a more declarative style for expressing the arguments:

import { Config } from "@pulumi/pulumi";

export const config = new Config<{ region: string, createDomain?: boolean }>("program", {
    region: "the region to deploy into",
    createDomain: {
        default: false,
        description: "if true, create a DNS entry",
    },
});

[joeduffy]

Here's another idea.

What if we put the config metadata inside of the Pulumi.yaml file. This would not only give us the above properties, while also eliminating the need to manually specify the name as part of the bag.

For example, in Pulumi.yaml:

name: program
config:
    region:
        description: The region to deploy into.
    createDomain:
        type: boolean
        default: false
        description: If true, create a DNS entry.

and then inside your program, simply point us to the Pulumi.yaml file:

export const config = new Config<{ region: string, createDomain?: boolean }>("./Pulumi.yaml");

The type argument there is optional but gives you good typing inside of TypeScript. Obviously, there would be no such thing in JavaScript:

export const config = new Config("./Pulumi.yaml");

I also think some simple shortcuts could make this even easier.

First, if there's no file part in the path, assume a Pulumi.yaml file part:

export const config = new Config("./");

And, if you omit the file path altogether, just assume the Pulumi.yaml is in the current directory:

export const config = new Config();

[lukehoban]

I like this latest proposal. A bit of a shame to have to say everything twice in different places, but feels like a pay-for-play way to add this extra metadata.

There's also a nice symmetry with Pulumi.stack.yaml holding values for these config sections. The naming of this file didn't really make sense previously, but starts to make more sense if the config is specified in Pulumi.yaml.

Also - what's the scenario that requires specifying where to get the config? Why not just always use Pulumi.yaml for consistency (and so that the code isn't required at all to understand what is the applicable config)?

[joeduffy]

Great points. This feels like a strictly better place than we are.

The only point where I hesitate is that it definitely goes against our "it's just code" philosophy; but given that config is the one place where it isn't, in fact, just code, this feels right to me.

what's the scenario that requires specifying where to get the config

I definitely wasn't imagining supporting or suggesting a different file than Pulumi.yaml. I was mostly imagining the case where you had a module like this

/
    config/
        index.ts *config is defined here*
    index.ts
    Pulumi.yaml

In this case, you'd need to say new Config("../"). We do this in the Terraform providers because we end up spitting out multiple config modules. I suppose we could do something like search the current directory and, if not found, keep searching upwards until we find a Pulumi.yaml? That at least doesn't give you the impression you can pass an arbitrary YAML file here.

Note that this is currently M16, post release. If we like this general path, we may want to change this.

[ellismg]

I suppose we could do something like search the current directory and, if not found, keep searching upwards until we find a Pulumi.yaml

Just to confirm that I understand this correctly. I assume that each Pulumi package now brings its own Pulumi.yaml with it (where as previously it did not?) and that we'd use this same sort of technology in the providers themselves (e.g. @pulumi/aws node package as a Pulumi.yaml in the root, and it does similar things to get at configuration? In this world, I assume we have to do special work so the partial paths are relative to their correct locations within the node_modules folder itself?

[justinvp]

We want to align with the direction we're heading for configuring PaC policies.

[Frassle]

During discussions there was a suggestion to generate strongly typed config classes from type descriptions in Pulumi.yaml.

That is given something like the above:

name: program
config:
    region:
        description: The region to deploy into.
    createDomain:
        type: boolean
        default: false
        description: If true, create a DNS entry.

We'd have a cli command that would spit out the right code for it, something like pulumi config generate that would look at the language for the current project (same as import) and write out a Config.ts/cs/java/go/etc file with something like (for TS):

import { Config } from "@pulumi/pulumi";

const _config = new Config()

export interface MyProjectConfig {
    region: string
    createDomain: boolean
}

export const config : MyProjectConfig = {
    region: _config.require("region"),
    createDomain: _config.getBoolean("createDomain") ?? false,
}

[ringods]

@Frassle I just submitted a PR for the Pulumiverse Github infrastructure code where I deliberately not used config.requireObject because it doesn't validate the config structure. See here for more of my explanation:

https://github.com/pulumiverse/infra/pull/13#issuecomment-1147625186

[dixler]

Just stashing this here as it's a weekend. I like that config is just config in https://github.com/pulumi/pulumi/issues/1052#issuecomment-374260067.

During discussions there was a suggestion to generate strongly typed config classes from type descriptions in Pulumi.yaml.

That is given something like the above:

name: program
config:
    region:
        description: The region to deploy into.
    createDomain:
        type: boolean
        default: false
        description: If true, create a DNS entry.

We'd have a cli command that would spit out the right code for it, something like pulumi config generate that would look at the language for the current project (same as import) and write out a Config.ts/cs/java/go/etc file with something like (for TS):

import { Config } from "@pulumi/pulumi";

const _config = new Config()

export interface MyProjectConfig {
    region: string
    createDomain: boolean
}

export const config : MyProjectConfig = {
    region: _config.require("region"),
    createDomain: _config.getBoolean("createDomain") ?? false,
}

Typed Config is good, but I think that codegen'ing the config is overkill and introduces more complexity since someone might already have a file called config.(ts|js|py|go|cs|java|etc.) and we've already reserved Pulumi.stack.yaml.

On a related topic, having a method on a provider that checks if the program's providers are configured and could let pulumi prompt users to complete missing information with something like pulumi configure or on running pulumi up, would be really nice.

[Frassle]

Typed Config is good, but I think that codegen'ing the config is overkill and introduces more complexity since someone might already have a file called config.(ts|js|py|go|cs|java|etc.) and we've already reserved Pulumi.stack.yaml.

The current intention is that codegening would be an option, not a requirement.

On a related topic, having a method on a provider that checks if the program's providers are configured and could let pulumi prompt users to complete missing information with something like pulumi configure or on running pulumi up, would be really nice.

We probably won't try to tackle provider config for the first version of this feature. The idea is being run past the provider team, but the expectation is the cost won't be worth it (at least for now).

[dannielshalev]

in our use case, we have hundreds of stack files per env (prod/stage/test) if we contain all of these files in a flat folder structure it may be hard to maintain and also prone to human error.

so very important requirement is to create an env hierarchy inside the project per env

a while back we worked with@phillipedwards and created https://github.com/phillipedwards/pulumi-custom-config

We took this raw code and placed it in our internal pulumi components usage with some minor modifications

If you can incorporate this code and capabilities inside your official pulumi Config() class it would be greatly appreciated.

[jkodroff]

I believe this is a pre-requisite to self-service infrastructure (service catalog) where users would fill out both required and optional config values from a UI.

Any reason we couldn't do something like const allConfig = Pulumi.Config.fromSchemaFile('path/to/file');? It's a single call, we can specify validation rules (again, necessary for a UI), and it would give the user all missing required values at once.

Note that AWS Proton uses an OpenAPI schema doc in YAML for this purpose: https://docs.aws.amazon.com/proton/latest/userguide/ag-schema.html

[EronWright]

Regarding the optionality of a given config key, I would argue that it should be possible to mark a key as optional without supplying a default value. Seems like that's not possible as of 3.87.1; validation fails if a configuration key has no value (and no default). Yet a program may well distinguish those cases (e.g. using Try).