SPIKE: Manage secrets in one place

[davidkelliott]

User Story

Currently we are managing secrets for workflows in 2 places, in secrets manager, then we are putting in them into Github repository secrets via our Github terraform code.

If the environments TF fails we can end up with no secrets in Github as seen here - https://mojdt.slack.com/archives/C013RM6MFFW/p1686913462629909?thread_ts=1686911210.327769&cid=C013RM6MFFW

This spike is to see if we could get them directly from secrets manager instead, since we already authenticate via OIDC so we don't use creds for that.

Timebox for 3 days but for 1 workflow - create a test workflow and running the plan

User Type(s)

MP engineers

Value

More resilient platform

Assumptions / Hypothesis / Questions / Unknowns

Proposal

At the beginning of a job, get the secrets from secrets manager

Unknowns

Not been tried before

Definition of done

• [ ] spike completed
• [ ] presented to team
• [ ] another team member has reviewed

Reference

How to write good user stories

[SimonPPledger]

https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_github.html

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity.

[richgreen-moj]

The issue of the secret not being updated in GitHub has happened once more where an account deletion was involved.... https://mojdt.slack.com/archives/C013RM6MFFW/p1710181133706049?thread_ts=1710158699.597139&cid=C013RM6MFFW Could be worth investing time in this ticket to avoid more occurrences

[sukeshreddyg]

We cannot pass the secrets from one job to another job (Not only secrets , but anything, if masked, cannot be referenced into another job). To ensure the proper management of secrets exclusively through AWS Secrets Manager, we need to implement the following solution:

Job1:

1. Retrieve the secrets from Secrets Manager using aws-actions/aws-secretsmanager-get-secrets@v2.
2. Implement encryption for a file by echoing those secrets into it.
3. Upload the file generated in step 2 to artifacts.

      - name: Get Credential
        id: get-secrets
        uses: aws-actions/aws-secretsmanager-get-secrets@v2
        with:
          secret-ids: |
            MODERNISATION_PLATFORM_ENVIRONMENTS,test

      - name: Encrypt the File
        run: |
          echo $MODERNISATION_PLATFORM_ENVIRONMENTS | gpg --quiet --symmetric --cipher-algo AES256 --batch --yes --passphrase '${{ secrets.PASSPHRASE }}' --output environment.txt.gpg
      
      - name: Upload the File
        uses: actions/upload-artifact@v4
        with:
          name: modernisation
          path: |
            environment.txt.gpg
          retention-days: 1

Job2:

In the subsequent job where you intend to utilize the secrets:

1. Download the artifacts generated in the previous job (using the same artifact name).
2. Decrypt the secrets for usage.

      - name: Download the File
        uses: actions/download-artifact@v4
        with:
          name: modernisation
          
      - name: Prepare for tests
        id: prepare
        #shell: pwsh
        run: |
          gpg --quiet --batch --yes --decrypt --passphrase='${{ secrets.PASSPHRASE }}' --output environment.txt environment.txt.gpg
          environment_management=`cat environment.txt`
          echo "::add-mask::$environment_management"
          echo "environment_management=$environment_management" >> $GITHUB_ENV

[sukeshreddyg]

Completed spike for fetching secrets from Secrets Manager. Confirmed feasibility for scheduled baseline workflow. Initiating ticket #6627 for testing integration. Next steps: test retrieval, update docs, ensure seamless integration for enhanced security