(integ-tests-alpha,custom-resources): snapshots are always outdated

[kornicameister]

Describe the bug

After upgrading to 2.146.0 from 2.145.0 where https://github.com/aws/aws-cdk/issues/30418 was introduced I am observing that running integration tests via integ-runner always fails. Snapshot is being reported as outdated, like on screenshot above. Although I can successfully rebuild/rerun tests and I can observe a change in snapshots next time I run tests I once again observe situation from screenshot

Once I had downgraded to 2.145.0 on both aws-cdk-lib, @aws-cdk/integ-runner and @aws-cdk/integ-tests-alpha to 2.145.0 tests are passing without a problem.

Behavior suggests a regression of sort around: https://github.com/aws/aws-cdk/issues/30418 Also the problem occurs, I believe, around custom resources I am using.

Expected Behavior

Screenshot are deterministically stored and next checks aren't failing.

Current Behavior

Reproduction Steps

N/A

Possible Solution

N/A

Additional Information/Context

N/A

CDK CLI Version

2.146.0

Framework Version

No response

Node.js Version

18.16.0

OS

MacOS sierra

Language

TypeScript

Language Version

5.4.5

Other information

No response

[kornicameister]

Hmm.. I thought I had fixed by downgrading to 2.145.0 but now I am running a test that has changed custom resource and it fails with same reason. This time going down to 2.144.0 fixes the problem consistently.

[pahud]

Yes I think https://github.com/aws/aws-cdk/pull/30418 would make your snapshot need to update as it makes logApiResponseData: true to logApiResponseData: undefined. This is a bug fix and existing snapshots would need to update.

[kornicameister]

@pahud but they are always updating and never actually doing that.

[TheRealAmazonKendra]

I'm updating a massive amount of our integration tests and experiencing this same thing. It looks like this is being injected at the time the test runs instead of at the build time. It's resulting in a good portion of our tests only working with the --dry-run flag being added, which we absolutely do not want. We want people running the the tests. I'm changing this to a p1.

[mrlikl]

I am experiencing this issue while running the test aws-stepfunctions-tasks/test/emrcontainers/integ.start-job-run.js. The test need to be run once again due to a change that I did. But during the update there is another change "logApiResponseData":true being set for the AwsCustomResource. Resulting in the lambda being invoked however, the AwsCustomResource only has onCreate thus the lambda is sending back empty data {}.

And the stack is failing with the error

CustomResource attribute error: Vendor response doesn't contain virtualCluster.containerProvider.id attribute in object arn:aws:cloudformation:us-east-1:xxxxxxxxxx:stack/aws-stepfunctions-tasks-emr-containers-start-job-run/7937d670-4289-11ef-98fb-0e5f2c0b2977|StartaJobRunGetEksClusterInfoD0E31373|eea731e7-2a04-4fe5-9947-d42d710d4e4f

Are there any work-arounds to not set the property logApiResponseData ? I have tried setting LOG_API_RESPONSE_DATA_PROPERTY_TRUE_DEFAULT to false in the test, however there has been no effect.

[kornicameister]

@TheRealAmazonKendra any news here?

[TheRealAmazonKendra]

I thiiiiiiiiiink I actually have figured out a piece of this. Unfortunately, my capacity has been extremely limited of late so I haven't had a chance to properly dig in. I'm hoping to do so a bit more next week because this is a huge pain.

[kornicameister]

@pahud @TheRealAmazonKendra @mrlikl this is an increasing problem for me. I'd very much like to update from 2.144.0 that I am stuck with to get ahold of some new developments but those tests are a blocker.

Did you by any chance found a fix, a workaround? What about https://github.com/cdklabs/aws-cdk-notices/blob/3fa7b639d8a19f5b7de773fb24c6c5cd43f8850b5/data/notices.json#L588, could that help?

[github-actions[bot]]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.

[iliapolo]

Reopening because the mentioned fix was reverted.

[mrgrain]

@kornicameister can you try the latest version of integ-runner v2.188.3?

You can then also see if the the new preview engine makes a difference: integ-runner --unstable=toolkit-lib-engine

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.