AWS Lambda .NET Core Deployment does not always make use of supplied AWS Credentials

[svisagie]

We've found that the AWS Lambda .NET Core Deployment task does not always make use of credentials in the supplied AWS Credentials. The credentials contains just an access id and secret access key, no role to assume.

We have a pipeline containing releases to multiple environments hosted in different AWS accounts using different credentials for each one.

The problem shows up in the logs that the connect S3 bucket region cannot be detected due to AccessDenied and then also later down when the the dotnet lambda package-ci process tries to upload the file to S3 it also fails due to an access denied error.

I've had a look through the NetCoreDeployTaskOperations.ts file and the problem seems to be related to what happens around line 40 where the following is found;

const env = process.env;

        // If assume role credentials are in play, make sure the initial generation
        // of temporary credentials has been performed. If no credentials were defined
        // for the task, we assume they are already set in the host environment.
        const credentials = await this.taskParameters.getCredentials();
        if (credentials) {
            env.AWS_ACCESS_KEY_ID = credentials.accessKeyId;
            env.AWS_SECRET_ACCESS_KEY = credentials.secretAccessKey;
            if (credentials.sessionToken) {
                env.AWS_SESSION_TOKEN = credentials.sessionToken;
            }
        }

        const region = await this.taskParameters.getRegion();

the last line is what throws the first error stating:

Warning: Unable to determine region for bucket BUCKETNAME, assuming bucket is in correct region: Access Denied

I think this is due to the fact that the dotnet lambda package-ci tool is not explicitly provided with the credentials supplied in the AWS Credentials setting, but rather relying of environment variables.

I've mostly managed to get around the issue by adding an AWS Powerhsell task before this task which has the same credentials and region specified and it just does the default Hello World message. This workaround doesn't always work and the deploy sometimes needs to be run for a second time for the credentials to actually take.

The issue is not permissions since this same AWS Credentials is used in several other places without any issue to push packages to S3 for CodeDeploy.

[stevejroberts]

If you are using fixed access and secret keys, and not assumed roles, then the pending fix for having the task await credential generation (issue #126) should not apply here.

Are you sure that the bucket you specified exists in the same region as the Lambda function you are creating/deploying to?

[svisagie]

100% sure the bucket exists and the credentials has access to it -- as stated above it does work from time to time, but since the credentials are set to environment variables (essentially global variables) on the machine which has multiple VSTS agents running at the same time using different credentials chances are pretty good that something else is going to overwrite the env.AWS_ACCESS_KEY_ID variable.

I would suggest to rather pass credentials.secretAccessKey and credentials.sessionToken as parameters to the command instead of setting env.AWS_ACCESS_KEY_ID and hoping that nothing else changes it

[stevejroberts]

My understanding is that agent builds run in separate processes so I hope there's no chance of an environment variable overwrite from parallel builds. I've not verified, beyond a cursory glance at the source code, that our dotnet cli extensions accept keys and token as parameters so it might not be possible anyway but I'll see what might be possible.

[svisagie]

OK, fair point regarding separate processes having their own environment variables. I will try to get my own custom version of the tools built containing a bit of extra logging to confirm what exactly the access id, etc is and will give feedback here if I'm successful.

[svisagie]

Hi @steveataws, I've not been able to create a custom version of the VSTS tools yet, but I've spent some more time investigating and also working through the actual DotNet Lambda Tools which the VSTS tools use.

I've found that there are definitely command line parameters that can be use to pass the access key and secret through (--aws-secret-key and --aws-session-token) -- common options can be found here (https://github.com/aws/aws-extensions-for-dotnet-cli/blob/master/src/Amazon.Common.DotNetCli.Tools/Options/CommonDefinedCommandOptions.cs).

I also found that when I use these in a manual test everything works perfectly, which at least confirms to me that my IAM permissions are correct. This still leads me to believe that there is some issue between the environment variables that are set in the VSTS tools and what is used in / passed to the DotNet Tools.

The DotNet Tools seems to have a very broad and complicated way of figuring out what credentials to use and from where (4 or 5 different ways) if the command line parameters aren't used. I suspect it finds and uses something else before it gets to the environment variables the VSTS task specifies which we want it to use. So I still think that use command line parameters would be much better.

[svisagie]

Found a more reliable way of working around the issue:

On the AWS Lambda .NET Core Deployment task I'm adding the following under Additional Lambda Tools Command Line Arguments : --aws-access-key-id $(aws-access-key-id) --aws-secret-key $(aws-secret-key), with aws-access-key-id and aws-secret-key added as secure variables.

This is still not 100% desirable since I now have to define/configure credentials in two places, but at least we can reliable deploy every time.

[stevejroberts]

Taking a look.

[stevejroberts]

Just to follow up, I looked into the possibility of switching over to using the credentials in command line parameters and this will only work where the global tools version of our dotnet CLI extensions are installed/in-use. The versions of the tools prior to them becoming 'global enabled' don't have these command line options.

We'll need to implement issue #113 first, then we can update this task so that if global tools are available it uses command line parameters otherwise it falls back to passing credentials in environment variables for backwards compatibility.

[svisagie]

Hi @steveataws

Any idea when #113 will be completed so that this issue can be resolved.

[svisagie]

Hi @steveataws, it's been over a year now, issue #113 has been completed for some time now and the workaround that I'm having to do because of this bug is blocking me from implementing the automating of IAM key rolling between Azure Devops and AWS

[rli]

The closure of #113 includes adding the session token as an environment variable to the Lambda .NET Core deployment task (https://github.com/aws/aws-toolkit-azure-devops/blob/master/Tasks/LambdaNETCoreDeploy/TaskOperations.ts#L53)

Does that solve your problem or is this an ask for passing credentials as command line parameters for all the dotnet tools across the board?