Cannot use runtime variable reference for awsCredentials

[athielen2]

Describe the bug I am attempting to use a runtime variable reference to specify the name of the AWS service connection based on the branch being built. When the pipeline is run, it fails automatically with the following error:

The pipeline is not valid. Job build_service_client_review_notes: Step AWSShellScript input awsCredentials references service connection $(aws_service_connection) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.

My template is defined as follows:

parameters:
- name: 'serviceDirectory'
  type: string
- name: 'serviceName'
  type: string
- name: 'jobId'
  type: string 

jobs:
- job: 'build_service_${{ parameters.jobId }}'
  displayName: 'Build, Test, and Package Service - ${{ parameters.serviceName }}'
  pool: 'xxxx'
  container: 'xxxx'
  variables:
    is_ci_build: $[in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI')]
    package_dir: 'dist'
    aws_region: 'us-east-2'
    aws_service_connection: 'AWSDevServiceConnection'
  steps:
  - task: ShellScript@2
    displayName: 'Configure build variables'
    inputs:
      scriptPath: '.devops/scripts/configure-build-environment.sh'

  - template: '/.devops/templates/steps/build-and-test-project.yml'
    parameters:
      projectDirectory: ${{ parameters.serviceDirectory }}
      projectName: ${{parameters.serviceName}}

  - script: |
      sls package --stage $(env) --package $(package_dir)
    displayName: 'Package'
    workingDirectory: ${{ parameters.serviceDirectory }}
  
  - task: AWSShellScript@1
    displayName: 'Deploy'
    inputs:
      awsCredentials: $(aws_service_connection)
      regionName: $(aws_region)
      scriptType: 'inline'
      disableAutoCwd: true
      workingDirectory: ${{ parameters.serviceDirectory }}
      inlineScript: |
        sls deploy --package $(package_dir) --force
    condition: and(succeeded(), or(eq(variables['force_deploy'], true), and(eq(variables['is_ci_build'], true), eq(variables['should_sls_deploy'], true))))

The variables is static right now, but the hope is to dynamically set it in a script to the correct environment based on the branch.

Any guidance would be appreciated. Thanks!

To reproduce

1. Create a pipeline that uses the template above, and replace the aws_service_connection variable with the name of a valid service connection from your account.
2. Run a build
3. Observe the error occur

Expected behavior

The runtime variable expression is evaluated at runtime.

Screenshots

Your Environment Using Azure DevOps at https://dev.azure.com

[hunterwerlla]

This should definitely work, see this issue: https://github.com/aws/aws-vsts-tools/issues/303 . Potentially the fix is to use ${{ }} instead of $()

[athielen2]

Correct, it works using the template expression syntax, but that limits how you can define the variable since it is processed at compile time instead of runtime. Therefore, I cannot set the connection based on a variable set in a previous step.

[hunterwerlla]

This seems to be an issue with how Azure DevOps works with connection strings. An alternative you can try is setting the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY which should be picked up by the task

[thinear-github-repo]

This isn't going to work when assuming roles, which the service connection allows you to do. It would require using the aws cli to assume the role. Any update on when this will be fixed?

[hunterwerlla]

@thinear-github-repo Hello, you can set AWS_ROLE_ARN to assume a role

[Fitmavincent]

I have something like this:

- task: AWSShellScript@1
      inputs:
        awsCredentials: 'azure-infra-deploy-dev'
        regionName: 'ap-southeast-2'
        scriptType: 'inline'
        workingDirectory: $(Build.SourcesDirectory)/projectx-infrastructure/
        inlineScript: |          
          ls -la
          pwd
          echo "Running validations"
          cdk synth project-infra-dev -o out
          cd out
      
   displayName: 'Validating AWS CDK output Development Account'

drwxr-xr-x  4 vsts docker 4096 May 20 04:36 .
drwxr-xr-x  6 vsts docker 4096 May 20 04:36 ..
drwxr-xr-x  4 vsts docker 4096 May 20 04:36 projectx
drwxr-xr-x 11 vsts docker 4096 May 20 04:37 projectx-infrastructure
/home/vsts/work/1/s

The AWS Shell not running inside the projectx-infrastructure even I specify the working directory.

[nitinnbisht]

I have something like this:

- task: AWSShellScript@1
      inputs:
        awsCredentials: 'azure-infra-deploy-dev'
        regionName: 'ap-southeast-2'
        scriptType: 'inline'
        workingDirectory: $(Build.SourcesDirectory)/projectx-infrastructure/
        inlineScript: |          
          ls -la
          pwd
          echo "Running validations"
          cdk synth project-infra-dev -o out
          cd out
      
   displayName: 'Validating AWS CDK output Development Account'

drwxr-xr-x  4 vsts docker 4096 May 20 04:36 .
drwxr-xr-x  6 vsts docker 4096 May 20 04:36 ..
drwxr-xr-x  4 vsts docker 4096 May 20 04:36 projectx
drwxr-xr-x 11 vsts docker 4096 May 20 04:37 projectx-infrastructure
/home/vsts/work/1/s

The AWS Shell not running inside the projectx-infrastructure even I specify the working directory.

This seems to be a different issue. Please try again with disableAutoCwd: true. The revised pipeline template should be like

- task: AWSShellScript@1
      inputs:
        awsCredentials: 'azure-infra-deploy-dev'
        regionName: 'ap-southeast-2'
        scriptType: 'inline'
        disableAutoCwd: true
        workingDirectory: $(Build.SourcesDirectory)/projectx-infrastructure/
        inlineScript: |          
          ls -la
          pwd
          echo "Running validations"
          cdk synth project-infra-dev -o out
          cd out
      displayName: 'Validating AWS CDK output Development Account'