claudia
claudia copied to clipboard
claudiajs
ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:
In our project, Lambda was last deployed successfully by CI with claudia 2021-09-14 ~16:17 CET. There have been no issues earlier.
A next attempt by CI, at 2021-09-15 ~1649 CET failed. Retry attempts failed. Manual attempts via CLI failed. Manual upload, publish, and creation of an alias, worked via the console (but no working version was produced because there was no investment in getting the package right).
Nothing of relevance was changed (always a strong statement, I know). There was no update of claudia, or related
packages between 2 deploys. A retry of the successful deployed version failed too.
Retries 2021-09-16 ~ 10:10 CET failed again.
The reported error is always the same:
updating configuration lambda.setupRequestListeners
ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:eu-west-1:NNNNN:function:XXXXXXXX
at Object.extractError (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
at Request.callListeners (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at Request.emit (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at callNextListener (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
at IncomingMessage.onEnd (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/event_listeners.js:313:13)
at IncomingMessage.emit (events.js:412:35)
at IncomingMessage.emit (domain.js:470:12)
at endReadableNT (internal/streams/readable.js:1317:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21) {
code: 'ResourceConflictException',
time: 2021-09-16T08:19:05.924Z,
requestId: 'cf98db8a-0457-4f92-9a68-19b37f326508',
statusCode: 409,
retryable: false,
retryDelay: 45.98667333028396
}
But this happens quick, before the package is build, or after.
We've felt for a while that claudia does some things twice, first checking, and then doing. When the error appears
late, we see several mentions of lambda.setupRequestListeners
loading Lambda config
loading Lambda config sts.getCallerIdentity
loading Lambda config sts.setupRequestListeners
loading Lambda config sts.optInRegionalEndpoint
loading Lambda config lambda.getFunctionConfiguration FunctionName=XXXXXXXX
loading Lambda config lambda.setupRequestListeners
packaging files
packaging files npm pack -q /opt/atlassian/pipelines/agent/build
packaging files npm install -q --no-audit --production
[…]
validating package
validating package removing optional dependencies
validating package npm install -q --no-package-lock --no-audit --production --no-optional
[…]
validating package npm dedupe -q --no-package-lock
updating configuration
updating configuration lambda.updateFunctionConfiguration FunctionName=XXXXXXXX
updating configuration lambda.setupRequestListeners
updating configuration lambda.updateFunctionConfiguration FunctionName=XXXXXXXX
updating configuration lambda.setupRequestListeners
ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:eu-west-1:NNNNNNNN:function:XXXXXXXX
at Object.extractError (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
at Request.callListeners (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at Request.emit (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
at callNextListener (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
at IncomingMessage.onEnd (/opt/atlassian/pipelines/agent/build/node_modules/claudia/node_modules/aws-sdk/lib/event_listeners.js:313:13)
at IncomingMessage.emit (events.js:412:35)
at IncomingMessage.emit (domain.js:470:12)
at endReadableNT (internal/streams/readable.js:1317:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21) {
code: 'ResourceConflictException',
time: 2021-09-16T08:19:05.924Z,
requestId: 'cf98db8a-0457-4f92-9a68-19b37f326508',
statusCode: 409,
retryable: false,
retryDelay: 45.98667333028396
}
Resources on the internet are barely any help.
AWS Lambda - Troubleshoot invocation issues in Lambda
mentions ResourceConflictException, but with a different message, and refers to VPCs, which we are not using.
UpdateFunctionConfiguration,
PublishVersion,
UpdateFunctionCode and others mention more
generally:
ResourceConflictException
The resource already exists, or another operation is in progress.
HTTP Status Code: 409
Other resources are no help:
- https://discuss.hashicorp.com/t/problem-updating-aws-lambda-function/20597
- https://stackoverflow.com/questions/58971446/resourceconflictexception-the-function-could-not-be-updated
Terraform Error publishing version when lambda using container updates code #17153 (Jan. 2021) mentions a "lock" / "last update status", which we can watch during execution using
> watch aws --profile YYYYYYY --region eu-west-1 lambda get-function-configuration --function-name XXXXXXXX
The output looks like
{
"FunctionName": "XXXXXXXX",
"FunctionArn": "arn:aws:lambda:eu-west-1:NNNNNNNNNNN:function:XXXXXXXX
"Runtime": "nodejs14.x",
"Role": "arn:aws:iam::NNNNNNNNNNN:role/execution/lambda-execution-XXXXXXXX",
"Handler": "lib/service.handler",
"CodeSize": 76984324,
"Description": "[…]",
"Timeout": 30,
"MemorySize": 2048,
"LastModified": "2021-09-16T08:19:05.000+0000",
"CodeSha256": "zQb6Vss0Zlug46HRjA8+bNe0i1TP6NWfrm70hC6zC90=",
"Version": "$LATEST",
"Environment": {
"Variables": {
"NODE_ENV": "production"
}
},
"TracingConfig": {
"Mode": "PassThrough"
},
"RevisionId": "9d5f5431-6f2f-4d39-9794-d86778b34446",
"Layers": [
{
"Arn": "arn:aws:lambda:eu-west-1:NNNNNNNNNNN:layer:chrome-aws-lambda:25",
"CodeSize": 51779390
}
],
"State": "Active",
"LastUpdateStatus": "Successful",
"PackageType": "Zip"
}
most of the time, but we see LastUpdateStatus change for a moment before the error occurs.
Terraform aws_lambda_function ResourceConflictException due to a concurrent update operation #5154 says, in 2018,
OK, I've figured out what's happening here based on a comment here: AWS has some sort of limit on how many concurrent modifications you can make to a Lambda function.
serverless 'Concurrent update operation' error for multi-function service results in both deployment and rollback failure. #4964 reports the same issue in 2018, and remarks:
I just heard back from AWS Premium Support, and they offered up a solution and the cause of the issue. It's not so much an issue with too many functions, as it is trying to do too many updates with a single function.
So, this appears to be a timing issue. Claudia should take it slower?
In a further attempt to get this working again, I added --aws-delay 10000 ---aws-retry 30 to the update command.
No joy. Fails just as fast.
Monitoring with
> watch -n1 aws --profile YYYYYYY --region eu-west-1 lambda get-function-configuration --function-name XXXXXXXX
this time showed no other LastUpdateStatus than "Successful". It might have happened in between polls.
LastModified and RevisionId did change once, but the CodeSha256 and CodeSize did not.
So something (the configuration?) did change, but the code did not.
Created branches, which only deploy in CI, for the 2 last successful deploys, which worked a gazillion times before.
Both experiments get the same problem now.
Since package-lock.json was not changed, and deploy happens with npm ci, this can only mean something at the AWS side changed, either in general, or in this particular instance.
We just ran into the same issue and found the reason here:
https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/
FYI: We upgraded the version of the aws provider in one of our terraform sets to the latest version and that seems to have cleared the problem.
@choeller thx for the response
That is consistent with our observations.
We just ran into the same issue and found the reason here:
https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/
FYI: We upgraded the version of the aws provider in one of our terraform sets to the latest version and that seems to have cleared the problem.
@spencer-aerian that cleared the problem in claudia ?!?
Or are you suggesting updating the version of the AWS SDK used inside claudia?
Dear Claudiajs,
Thx for creating and maintaining this project. We've been using it for deployment for a number of years now (and not other features).
This issue is blocking for us. There has been little activity here over the last few months. Can you give us an indication about your intentions with this project?
I would like to be able to determine whether it is worth waiting for further progress, or whether it is more appropriate to look for another long term solution.
FYI: We upgraded the version of the aws provider in one of our terraform sets to the latest version and that seems to have cleared the problem.
@spencer-aerian that cleared the problem in
claudia?!?Or are you suggesting updating the version of the AWS SDK used inside
claudia?
No different package but same error.
I just checked out the project, and ran the tests (this takes ages).
There are some errors.
Notably, there are frequent mentions of
error cleaning up TooManyRequestsException: Too Many Requests
so I guess these tests are leaving behind some flutsam in our AWS account now.
Also, there are failures with update / environment variables. As far as I can see, there are more keys in the object returned than expected. AWS_LAMBDA_INITIALIZATION_TYPE seems unexpected.
But the heart of this issue is The operation cannot be performed at this time. An update is in progress …. And that message is exactly what we get with tests that work with layers. All those tests fail with this message.
update / layer support all fail with this message.
I am running the tests with both the AWS-SDK defined in the project, and the most recent version, and they fail with the latest AWS-SDK. In other words, updating the AWS SDK is not the answer in this case (it was fairly recent to start with).
Getting this problem here, unable to deploy an update of my lambda in production
@gojko - Also, still getting this on v5.14.0 I am using api-gateway and have a temporary work around by creating the gateway under a different name (that works just fine). This just started happening for me today - I successfully updated yesterday. Also I just upgraded to latest aws-sdk v2.1031.0
Maybe irrelevant and not applicable, but I solved this by adding the env var to my Cloudformation template instead of during deploy.
Unrelated: I don't use this library, but looks like the code now has to waitFor() the previous function update before running further updates.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html
For example, first the function is created and then its configuration is updated — has to wait in-between. Same is for updating the function's *.zip file and then reconfiguring it — has to wait too.
log('Updating lambda function');
// Update lambda code
await lambdaApi.updateFunctionCode({
FunctionName: getFunctionName(lambdaConfig, stage),
ZipFile: zipFile
}).promise();
log('Updated code');
// Has to wait for the function update status to transition to "Updated"
// until making further re-configuration.
// https://github.com/claudiajs/claudia/issues/226
// https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/
// https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html
await lambdaApi.waitFor('functionUpdated', {
FunctionName: getFunctionName(lambdaConfig, stage)
}).promise();
// Update lambda configuration
await lambdaApi.updateFunctionConfiguration(lambdaConfiguration).promise();
log('Updated configuration');
Started happening to me today as well. updating to the latest ClaudiaJs and AWS-SDK didn't help.
I was however able to mitigate the error by adding the aws:states:opt-out in the description of the lambda function or by not passing the environment variables during deploy.
Started for us as well. if someone wants to do quick fix until its fixed in ClaudiaJs, use below around Claudia commands:
aws lambda update-function-configuration --function-name ${FuncName} --description "aws:states:opt-out"
claudia update
aws lambda update-function-configuration --function-name ${FuncName} --description ""
I started making skills for Alexa for about a month. I was doing that using ASK CLI. This morning this error popped out:
[Error]: CliError: The lambda deploy failed for Alexa region "default": ResourceConflictException: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:454044463163:function:ask-reading-buses-default-default-1636455775666
Now I can't deploy any skill.
I all of a sudden had the same problem using the same program I have always used to deploy. I was updating the lambda code followed by a publish version. I would get the error "An update is in progress for resource". I fixed it by calling the getFunctionConfiguration and waiting til State was Active and LastUpdateStatus was Successful. I then continued on with the publish. The State was Active right away, but now it took the LastUpdateStatus to be Successful about 30 seconds or so.
I was in trouble with the same error on Node 12. Just update the Node version to 14 and it's working for me.
It looks like this is only still an issue if updateConfiguration is called with any options.
For me, removing the --runtime argument allowed the function to deploy correctly.
@gojko Something similar to the following should be enough to fix this (just copied the existing wait logic up into the updateConfiguration function)
--- a/src/commands/update.js
+++ b/src/commands/update.js
@@ -145,7 +145,11 @@ module.exports = function update(options, optionalLogger) {
},
() => logger.logStage('waiting for IAM role propagation'),
Promise
- );
+ ).then(result => {
+ logger.logStage('waiting for lambda resource allocation');
+ return waitUntilNotPending(lambda, lambdaConfig.name, awsDelay, awsRetries)
+ .then(() => result);
+ });
}
},
cleanup = function () {
https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/
This is still broken for me as well. I temporarily bypassed the situation by adding the opt out clause (aws:states:opt-out) to the Lambda description field, but this trick will only work until December 6th.
@magoyo Same here, However, i using code-build which is similar. So, the solution is add --description "aws:states:opt-out" for any update?
@ambigus9 I didn't see a --description option in the update documentation (it was only on create), so I added it to the project description in package.json and also to the description field in Lambda configuration screen. It worked like a charm, but it's obviously temporary and we don't have much time.