Cannot deploy SRE

[jemrobinson]

:white_check_mark: Checklist

• [x] I have searched open and closed issues for duplicates.
• [x] This is a problem observed when deploying a Data Safe Haven.
• [x] I can reproduce this with the latest version.
• [x] I have read through the documentation.
• [x] This isn't an open-ended question (open a discussion if it is).

:computer: System information

• Operating System: macOS
• Data Safe Haven version: develop @ c1d0adc

:no_entry_sign: Describe the problem

I cannot deploy an SRE (after successfully deploying an SHM) as the encryption key is not copied from the SHM project into the SRE project.

Possibly encryption key should be a top-level property of the DSHPulumiConfig instead of belonging to each DSHPulumiProject? Alternatively, each DSHPulumiProject should generate its own encryption key

:deciduous_tree: Log messages

Relevant log messages

2024-05-02 12:11:02 [   ERROR] Could not deploy Secure Research Environment apple.                                                                                                                                                                                   cli.py:99
2024-05-02 12:11:02 [   ERROR] Pulumi deployment failed.                                                                                                                                                                                                             cli.py:99
2024-05-02 12:11:02 [   ERROR] Applying Pulumi configuration options failed.                                                                                                                                                                                         cli.py:99
2024-05-02 12:11:02 [   ERROR] 1 validation error for DSHPulumiProject                                                                                                                                                                                               cli.py:99
2024-05-02 12:11:02 [   ERROR] encrypted_key                                                                                                                                                                                                                         cli.py:99
2024-05-02 12:11:02 [   ERROR]   Input should be a valid string                                                                                                                                                                                                      cli.py:99
2024-05-02 12:11:02 [   ERROR]     For further information visit https://errors.pydantic.dev/2.7/v/string_type.

:recycle: To reproduce

• Deploy an SHM
• Attempt to deploy an SRE

[JimMadge]

Possibly encryption key should be a top-level property of the DSHPulumiConfig instead of belonging to each DSHPulumiProject?

A single secrets provider is shared across all Pulumi projects and stacks? If so, that sounds right.

Alternatively, each DSHPulumiProject should generate its own encryption key

I would think that is what is happening now.

[jemrobinson]

It's currently created once per context here: https://github.com/alan-turing-institute/data-safe-haven/blob/develop/data_safe_haven/context/context.py#L82-L85.

[JimMadge]

Oh, I mean the encrypted_key value is per DSHPulumiProject.

(I'm still not entirely certain what that actually is).

[JimMadge]

By the way, encrypted_key is present in the state file <stack>.json. However, it doesn't seem you can read it from there, maybe parsing the file fails as there are encrypted secrets in it?

[jemrobinson]

This line (https://github.com/alan-turing-institute/data-safe-haven/blob/develop/data_safe_haven/infrastructure/project_manager.py#L108) loads the encryption key from a DSHPulumiProject, but there's nothing that sets the encryption key for a new DSHPulumiProject project (the default constructor from https://github.com/alan-turing-institute/data-safe-haven/blob/develop/data_safe_haven/config/pulumi.py#L61 sets this to a blank string).

The key could instead be stored at the top level, so instead of

projects:
  apple:
    encrypted_key: keyvalue
    stack_config:
      azure-native:location: uksouth
      ...
  green:
    encrypted_key: keyvalue
    stack_config:
      azure-native:location: uksouth
      ...

we have

encrypted_key: keyvalue
projects:
  apple:
    stack_config:
      azure-native:location: uksouth
      ...
  green:
    stack_config:
      azure-native:location: uksouth
      ...

Alternatively, if we want to store the encrypted_key for each project, we should ensure that this is passed in by the default constructor in config/pulumi.py.

[JimMadge]

I don't think we can pass it on the construction of a new DSHPulumiProject as don't necessarily know what it is.

It gets written here.

https://github.com/alan-turing-institute/data-safe-haven/blob/1cbb819e372dcb61a074e8740578c0dacf803635/data_safe_haven/infrastructure/project_manager.py#L363-L371

I could see writing it to the DSHPulumiConfig once (if it isn't defined) and always reading from there.