Restructure as Pulumi inline program

[JimMadge]

:strawberry: Suggested change

It may be possible to do away with local files/directory structure for Pulumi configuration and Pulumi stack configuration. The information in these files may be provided at run time as Python objects.

:steam_locomotive: How could this be done?

• https://github.com/pulumi/automation-api-examples/tree/main/python/inline_program
• https://www.pulumi.com/docs/using-pulumi/automation-api/concepts-terminology/#inline-program

[jemrobinson]

We're already doing this i.e. running as an inline program not a local program.

We still need the Pulumi YAML files in order to synchronise state between multiple users deploying/changing the same stack. Note that inline programs still make use of these files (but hide it from you), in particular:

• Pulumi.yaml which is only 3 lines long and says where you last ran Pulumi from and which language you're using
• Pulumi.<stack-name>.yaml which has:
  • secretsprovider (location of the Pulumi encryption key)
  • encryptedkey (something related to the value of the key - not simply its contents)
  • config which includes any secure values that need to be decrypted by Pulumi

These are the Base64-encoded files that we currently store in our shared Azure Storage backend.

[JimMadge]

Hmm, having a closer look at the docs, I think that is right. A local program only takes a name and work_dir. Although we are using inline in an odd way.

If you don't specify work_dir in opts then commands like create_or_select_stack will create a temporary directory and populate that with the configuration and stack settings. So it is odd to give a work_dir and write files there ourselves instead of just passing them,

stack=create_or_select_stack(
    stack_name,
    project_name,
    program = program,
    opts = LocalWorkspaceOpts(
        StackSettings(
            secrets_provider = ...
            encrypted_key = ...
            config = {...}
        )
   )
)

• https://www.pulumi.com/docs/reference/pkg/python/pulumi/#pulumi.automation.create_or_select_stack
• https://www.pulumi.com/docs/reference/pkg/python/pulumi/#pulumi.automation.StackSettings

[JimMadge]

I think I would be keen to try and do this all in one PR #1820. That way we don't have to worry about working directories or writing local files.

[jemrobinson]

Passing the values in and letting Pulumi sort out how to manage the local files is certainly reasonable. Let me know if it works!

NB. I don't think we ever store the value of encrypted_salt so you might need to add that to the storage backend too. Or maybe just use the _serialize and _deserialize methods of StackSettings and ProjectSettings?

[jemrobinson]

@JimMadge : is this closed by #1820?

[JimMadge]

Yes :smile: