jason-johnson
Terraform apply fails with clientCredentialsToken already expired
Describe the bug Terraform apply sometimes fails in DevOps pipelines using OIDC authentication due to expired credentials:
Error: error loading state: autorest/Client#Do: Preparing request failed: StatusCode=0 -- Original Error: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-03-28T13:09:23.9070719Z, assertion valid from 2024-03-28T12:38:32.0000000Z, expiry time of assertion 2024-03-28T12:48:31.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials .
This issue appears when an apply is run using a plan created more than 10 minutes earlier. For example when there is a manual approval step between plan and apply which took over 10 minutes, or when re-running a failed apply job after 10 minutes of creating the plan.
The root cause seems to be that terraform is saving the temporary OIDC credentials in the plan file, which is then attempted to be reused later, when the temporary credentials have expired. Issue for root cause: https://github.com/hashicorp/terraform/issues/34322
Similar issue in the MS DevLabs extension: https://github.com/microsoft/azure-pipelines-terraform/issues/201 PR for proposed workaround in MS DevLabs extension: https://github.com/microsoft/azure-pipelines-terraform/pull/214
I have verified that the known workaround of switch the task to AzureCLI@2 (and passing env vars for authentication) fixes the issue.
Is it possible to port this workaround to this extension?
To Reproduce Steps to reproduce the behavior:
- Setup DevOps service connections using User Assigned Managed Identities and workload identity federation (OIDC auth)
- Setup pipeline with regular terraform init, plan, apply steps. Put a ManualValidation@ job in between the plan and apply.
- Execute pipeline, wait > 10 minutes before manually approving and continuing with the apply
- See error
Expected behavior I expect the apply step to run.
Screenshots If applicable, add screenshots to help explain your problem.
Pipeline Logs Include logs that help demonstrate the problem. Please make sure to redact any sensitive info such as secrets.
Agent Configuration
- OS: ubuntu
- Self Hosted
- Terraform version used: 1.7.5
- AzureCLI version used: N/A
Additional context Add any other context about the problem here.
FYI @jaredfholgate Tagging you since I've noticed you are working on a workaround for this issue in the DevLabs extension.
Is there any chance to see this issue resolve soon or any workaround available?
The simplest workaround would be to have a step that does the plan, then after approvals run plan again, make sure it matches the previous and then run the apply afterwards. The workaround done in the DevLabs is a trade off that I'm not sure we can make.
One use case more to this issue. I am building the plan file in Azure Devops yaml pipelines (the CI part of process). Then it is transferred as an artifact to the Classic Release Pipelines (the CD part of process). I have all the variables etc prepared in the CI part so the plan should be good to go without running "plan" again in the CD part. I couldn't set the variables and all that in the Classic pipeline the same way it is done in the yaml pipelines.
Well, now the plan is not valid anymore after 10 minutes. And I can't run the plan, again, when I am applying it.
The only "fix" I know for this is described here. This would make the extension harder to use in various multi-subscription scenarios. I'm afraid this is a wont fix, at least for now. I can only suggest the workaround discussed above.