Errors Deploying to demo1

[wcutshall]

DevSecOps Issue

Summary

After merging origin/platform/bill/17644 to the demo1 branch the subsequent GitHub workflow that was initiated failed in the "Deploy Infrastructure" step with what appears to be issue with credentials and database instances.

Environment

• [x] demo1
• [ ] Local
• [ ] Dev
• [ ] Stage
• [ ] Prod

Priority

• [ ] Critical - affecting prod systems
• [x] Major - blocking major functionality, deployment, etc
• [ ] Minor - improvements, bug fixes
• [ ] Nice-to-have - feature request

Blocks the following

• Unable to test Dockerfile fixes for security patches.

Blocked by the following

• N/A

Contact

Platform:

Slack Channel: cdc-reportstream-platform-new

**** Bill Cutshall (AU81)

Tech Lead: OluOwoseni

How to Reproduce

Re-run from the GitHub page: https://github.com/CDCgov/prime-reportstream/actions/runs/14317736226

Screenshots, links, etc. for context

How to Test

Definition of Done

Able to deploy to the demo1 backend

• [ ]

Context Links

Git Repo: prime-reportstream

Relevant Code Links:

Azure:

Site URL: https://demo1.reportstream.cdc.gov/login

Other:

Notes

[wcutshall]

Console output from the process:

...

│ Error: making Read request on Azure KeyVault Secret functionapp-postgres-pass: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The user, group or application 'appid=***;oid=ac21f8dc-f2ee-4a56-bd88-51a5ae230dd8;iss=https://sts.windows.net/***/' does not have secrets get permission on key vault 'pdhdemo1-appconfigs5m;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287" InnerError={"code":"AccessDenied"}
│ 
│   with module.init.azurerm_key_vault_secret.init["functionapp-postgres-pass"],
│   on ../../modules/init/key_vault.tf line 140, in resource "azurerm_key_vault_secret" "init":
│  140: resource "azurerm_key_vault_secret" "init" {
│ 
╵
╷
│ Error: making Read request on Azure KeyVault Secret functionapp-postgres-user: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The user, group or application 'appid=***;oid=ac21f8dc-f2ee-4a56-bd88-51a5ae230dd8;iss=https://sts.windows.net/***/' does not have secrets get permission on key vault 'pdhdemo1-appconfigs5m;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287" InnerError={"code":"AccessDenied"}
│ 
│   with module.init.azurerm_key_vault_secret.init["functionapp-postgres-user"],
│   on ../../modules/init/key_vault.tf line 140, in resource "azurerm_key_vault_secret" "init":
│  140: resource "azurerm_key_vault_secret" "init" {
│ 
╵
╷
│ Error: making Read request on Azure KeyVault Secret sendgrid-password: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="The user, group or application 'appid=***;oid=ac21f8dc-f2ee-4a56-bd88-51a5ae230dd8;iss=https://sts.windows.net/***/' does not have secrets get permission on key vault 'pdhdemo1-appconfigs5m;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287" InnerError={"code":"AccessDenied"}
│ 
│   with module.init.azurerm_key_vault_secret.init["sendgrid-password"],
│   on ../../modules/init/key_vault.tf line 140, in resource "azurerm_key_vault_secret" "init":
│  140: resource "azurerm_key_vault_secret" "init" {
│ 
╵
Command completed after 1 attempt(s).
Run az postgres server update -g prime-data-hub-demo1 \
WARNING: Azure Database for PostgreSQL – Single Server is scheduled for retirement by March 28 2025, (https://go.microsoft.com/fwlink/?linkid=2300058). Migrate to Azure Database for PostgreSQL - Flexible Server now, (https://go.microsoft.com/fwlink/?linkid=2197657).
ERROR: (ResourceNotFound) The Resource 'Microsoft.DBforPostgreSQL/servers/pdhdemo1-pgsql-replica' under resource group 'prime-data-hub-demo1' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.DBforPostgreSQL/servers/pdhdemo1-pgsql-replica' under resource group 'prime-data-hub-demo1' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Error: Process completed with exit code 3.
Run sleep 300;
WARNING: Azure Database for PostgreSQL – Single Server is scheduled for retirement by March 28 2025, (https://go.microsoft.com/fwlink/?linkid=2300058). Migrate to Azure Database for PostgreSQL - Flexible Server now, (https://go.microsoft.com/fwlink/?linkid=2197657).
ERROR: (ResourceNotFound) The Resource 'Microsoft.DBforPostgreSQL/servers/pdhdemo1-pgsql' under resource group 'prime-data-hub-demo1' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.DBforPostgreSQL/servers/pdhdemo1-pgsql' under resource group 'prime-data-hub-demo1' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Error: Process completed with exit code 3.

[devopsmatt]

I don't have privileges to fix this, opened a ticket: Submitted :04/10/2025 13:04:39 Request Number : REQ0441195

[devopsmatt]

@wcutshall This should be working now, the Az permissions were fixed. Please merge origin/main into your branch to pick up TF changes before retrying this.

[devopsmatt]

referenced errors https://github.com/CDCgov/prime-reportstream/actions/runs/14619553674/workflow

[devopsmatt]

This is now no longer needed, since the demo environments are going away in O&M. The work will be continued in the context of test environment, which will replace the demo environments, in a new ticket.