bref icon indicating copy to clipboard operation
bref copied to clipboard

UPDATE or ROOLBACK FAILED in cloudFormation

Open cedric-r-mycelium opened this issue 3 weeks ago • 13 comments

Description:

Since monday my lambda can't be deployed ou rollbacked with the error "CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [...]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version"

I have open ticket at AWS support but the response is bad implementation of cnf-response module https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-lambda-function-code-cfnresponsemodule.html

i have try with

  • official serverless V3
  • and now on osls with the same result
  • bref in version 2.4.16

How to reproduce:

the probleme is on S3 custom resource in serverless.yml

 s3:
   existing: true
   forceDeploy: true
   bucket: "mybucket"
   event: s3:ObjectCreated:*
   rules:
      - prefix: photos/original
  

cedric-r-mycelium avatar Dec 04 '25 06:12 cedric-r-mycelium

I don't understand why the AWS support talks about cfn-response, this module doesn't appear anywhere in Bref or osls.

Could you confirm you tried on the latest version of osls?

Do you have any more details about the errors? Any logs in CloudWatch? (it seems to be a "custom resource that failed during the deployment -> a custom resource is actually a Lambda function written in Node that is called by CloudFormation to set up some infrastructure, so it should have some logs in CloudWatch like PHP functions)

Maybe that custom resource is on a Node version that is no longer supported? It could be worth checking in the .serverless/... what Node version is for that function, or you could check in the AWS console. I know there are discussions about upgrading Node function versions for custom resources right now in the osls repository, maybe that's it.

mnapoli avatar Dec 04 '25 09:12 mnapoli

osls version: 3.61.0

I have try to deploy from my dev EC2 in nodeJs 20 and nodeJs 24 but atfer a long waiting (3000s) i have only CREATE_FAILED - Custom::S3 - monbucket But for update it's code pipeline and codebuild for launch the sls deploy

Monday nodejs 20 Serverless V3 on code build Now nodejs 24 osls v3.61.0 on code build

cedric-r-mycelium avatar Dec 04 '25 13:12 cedric-r-mycelium

And we are on amazon linux 2 x86_64

arn:aws:lambda:eu-west-1:534081306603:layer:php-82:107 arn:aws:lambda:eu-west-1:403367587399:layer:redis-php-82:17 arn:aws:lambda:eu-west-1:403367587399:layer:igbinary-php-82:17

cedric-r-mycelium avatar Dec 04 '25 13:12 cedric-r-mycelium

After if you need more log , juste tell me how find them !!!

cedric-r-mycelium avatar Dec 04 '25 13:12 cedric-r-mycelium

I have test my code build and console sls deploy with osls with the PR https://github.com/oss-serverless/serverless/pull/114 it work fine for update 🍾 but for create new stack custom:S3 it's not working !!




INIT_START Runtime Version: nodejs:18.v96	Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:09b3ce2fcac5e073136005f4073a1a934831c7cf34c2d3054c7ece4474f4bee0
--
START RequestId: edd59128-c628-4a72-b6d2-191cf0c2f74b Version: $LATEST
2025-12-05T07:32:41.804Z	edd59128-c628-4a72-b6d2-191cf0c2f74b	INFO	{     "StackId": "arn:aws:cloudformation:eu-west-1:*:stack/new-stack/56c75710-d1ac-11f0-84a4-0ad64dccc96f",     "RequestId": "3389de87-cd43-4367-a232-b1b15460134b",     "LogicalResourceId": "GenPhotosS3CustomS31",     "PhysicalResourceId": "CustomResourceExistingS3",     "Status": "FAILED",     "Reason": "Unable to validate the following destination configurations See details in CloudWatch Log: 2025/12/05/[$LATEST]c2ab0a4fd09d474aa59bff6ca9c9f380",     "Data": {} }
2025-12-05T07:32:41.848Z	edd59128-c628-4a72-b6d2-191cf0c2f74b	INFO	STATUS: 200
2025-12-05T07:32:41.848Z	edd59128-c628-4a72-b6d2-191cf0c2f74b	INFO	HEADERS: {     "x-amz-id-2": "9KhyC1IgjIcQYYx15WSytaTbuTkUmsGqBG7sHulcxcaiybc+EI0S/xS151fhPcPdg9/x1KNu3cDybrllgh/tubxHrP/csoDZ",     "x-amz-request-id": "R8H240KVPP3Z528T",     "date": "Fri, 05 Dec 2025 07:32:42 GMT",     "x-amz-server-side-encryption": "AES256",     "etag": "\"b915b4db927b3e50f2500d26f07461e6\"",     "x-amz-checksum-crc64nvme": "SYoFoBeb788=",     "x-amz-checksum-type": "FULL_OBJECT",     "content-length": "0",     "server": "AmazonS3",     "connection": "close" }
END RequestId: edd59128-c628-4a72-b6d2-191cf0c2f74b




cedric-r-mycelium avatar Dec 05 '25 07:12 cedric-r-mycelium

I don't know why, but I deleted the problematic bucket and recreated it, and now it also works with new deployments.

cedric-r-mycelium avatar Dec 05 '25 13:12 cedric-r-mycelium

Can you try with https://github.com/oss-serverless/serverless/releases/tag/v3.61.1 if it helps?

I managed to reproduce with this: https://github.com/oss-serverless/serverless/issues/130

mnapoli avatar Dec 05 '25 14:12 mnapoli

Bad news, it's not working with the 3.61.1 :( but the cloudformation in not broken this time, it's rollback good

cedric-r-mycelium avatar Dec 08 '25 08:12 cedric-r-mycelium

This Custom::S3 resource is in the UPDATE_FAILED state.

CloudFormation did not receive a response from your custom resource. Please check your logs for request ID [20d11ec2-598a-4811-b43e-61aa2195bb51]. If you are using the cfn-response Python module, you may need to update your Lambda function code so that CloudFormation can associate it with the updated version.

cedric-r-mycelium avatar Dec 08 '25 08:12 cedric-r-mycelium

nothing test in progress

cedric-r-mycelium avatar Dec 08 '25 09:12 cedric-r-mycelium

i have try with you exemple #130 with nodejs16 it's not working but with nodejs22 it's ok

cedric-r-mycelium avatar Dec 08 '25 09:12 cedric-r-mycelium

CREATE a new stage ✅ ok UPDATE the new stage ✅ ok but UPDATE an old stage ❌not working (but it worked Friday afternoon with the patch branch)

cedric-r-mycelium avatar Dec 08 '25 10:12 cedric-r-mycelium

https://github.com/oss-serverless/serverless/issues/130

cedric-r-mycelium avatar Dec 09 '25 08:12 cedric-r-mycelium