(core): unused cross-region exports can't be deleted

[shahin]

Describe the bug

If a stack maintains multiple cross-region exports, and some subset of cross-region exports are being imported by other stacks, attempting to remove any cross-region export value that is not being used (imported) by any deployed stack:

❯ cdk diff [...] --exclusively
[...]
[~] Custom::CrossRegionExportWriter ExportsWriteruswest2[...]
ExportsWriteruswest2[...]
 └─ [~] WriterProps
     └─ [~] .exports:
         └─ [-] Removed: ./cdk/exports/unused-value/anotherstackuseast1Ref[...]certificate[...]

Fails with the following error on cdk deploy:

ExportsWriteruswest2[...]/Resource/Default (ExportsWriteruswest2[...]) Received response status [FAILED] from custom resource. Message returned: Error: Exports cannot be updated: 

    at throwIfAnyInUse (/var/task/index.js:4:10)

Other issues don't describe this bug:

• several closed issues reference cross-region exports (https://github.com/aws/aws-cdk/issues/24464, https://github.com/aws/aws-cdk/issues/25114), but these concern creating exports not deleting them
• many closed issues reference removing cross-stack exports (https://github.com/aws/aws-cdk/issues/7602), but cross-stack exports that are not also cross-region don't use the ExportsWriter lambda and can't produce the failure mode
• the Stack.export_value function does not export values across regions, only across stacks, so it can't be used to create a "dummy" cross-region export for this ExportsWriter lambda

Expected Behavior

CDK should fail to deploy with the error above only if the cdk diff shows an export being removed that is currently imported by another stack.

Current Behavior

CDK fails to deploy with the error above even when cdk diff shows only exports being removed that are not currently imported by another stack.

Reproduction Steps

1. Create two cross-region exports, exported from stack A into another stack B in a different region.
2. Remove one of the two imports from stack B and deploy stack B exclusively. Observe that one import is removed from stack B's ["CrossRegionExportReader"]["ReaderProps"] template in the CloudFormation console.
3. Attempt to deploy stack A. Observe one of the two exports being removed in the diff. Observe that on the deploy, the ExportWriter fails to update and the deploy fails with the error above.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.94.0 (build 987c329)

Framework Version

No response

Node.js Version

v18.17.1

OS

macOS

Language

Python

Language Version

Python (3.10.4)

Other information

No response

[khushail]

Hi @shahin , could you please share the sample code so that this issue can be reproduced

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[shahin]

Thanks for your attention to this issue. The reproduction steps above are the best I can do for now.

[renschler]

I am facing this issue as well. @shahin do you know if there is a workaround? Is manually deleting the SSM parameters in all regions before deploy a way to get around this? Im debugging in a dev environment now but trying to figure out how I'm going to roll my changes out to prod without a crash; makes me nervous haha

Reproduction Steps Create two cross-region exports, exported from stack A into another stack B in a different region. Remove one of the two imports from stack B and deploy stack B exclusively. Observe that one import is removed from stack B's ["CrossRegionExportReader"]["ReaderProps"] template in the CloudFormation console. Attempt to deploy stack A. Observe one of the two exports being removed in the diff. Observe that on the deploy, the ExportWriter fails to update and the deploy fails with the error above

My scenario was slightly different, but essentially the same issue.

• When I removed one of the two imports from stack B, and deployed stack A exclusively I got a crash. When I attempted to deploy the who app at once (ie both stacks at once) I also got a crash. (Note there is no way for me to deploy stack B exclusively from what I can tell because I've specified that stack B is dependent on stack A).

[shahin]

@renschler sorry, I don't know of a workaround in CDK. Deleting the SSM parameters might be part of the solution but did not by itself unblock deployment for us.

(Note there is no way for me to deploy stack B exclusively from what I can tell because I've specified that stack B is dependent on stack A).

If stack B depends on stack A, you can still deploy stack B by itself with the --exclusively flag:

❯ cdk deploy --help
cdk deploy [STACKS..]

[...]
-e, --exclusively          Only deploy requested stacks, don't include
                             dependencies                              [boolean]

[renschler]

thanks @shahin

@khushail I made a reproduction repo please let me know if you have any questions

https://github.com/renschler/cross-region-delete

[renschler]

ps as a warning to others, don't delete those unnecessary SSM paramters, it will create more issues! I'm now having to re-add them manually to SSM in every region where I deleted them. then after manually re-adding you'll hit a "last modified date does not match with the last modified date of the retrieved parameters" error during deploy, the only way to fix was to run cdk synth and then manually update the stacks in the aws console using the generated json template in cdk.out/

[NickDarvey]

Given the bug found by @elliotsegler in https://github.com/aws/aws-cdk/issues/29699, it looks like the error Error: Exports cannot be updated is explicitly thrown when an SSM parameter has the tag aws-cdk:strong-ref:SomeStackName ^1.

Perhaps a workaround is removing that tag and then redeploying? I understand the tag is there to stop updates ^2, but I'm not sure why that's necessary [^3]. (So this workaround might not be a very good workaround!) In my case I am making a change that removes a cross-region reference so... should be fine?

[^3]: My best bet is that CloudFormation won't detect a change in the parameter's value for the importing stack, so resources using the parameter won't actually get updated with the new value. The strong-ref check is there to stop people getting confused about why their resourced don't get updated with the new value.

[elliotsegler]

@NickDarvey I'm wondering if that's an ordering issue...

The cross-region-reader-handler and associated Custom::CrossRegionReader resource get added to the stack that consumes the exports. That handler is capable of removing the tags^1.

In my issues I was letting CDK resolve the order of operations, and it was attempting to delete resources in the stack that contained the exports before updating the reader stack which would have removed those tags. I worked around it by specifically calling a deployment of the reader stack, then a second phase of deployments to the stack that contained now unused exports.

[NickDarvey]

I worked around it by specifically calling a deployment of the reader stack

That would have worked for me too actually!

[renschler]

thanks @shahin

@khushail I made a reproduction repo please let me know if you have any questions

https://github.com/renschler/cross-region-delete

@TheRealAmazonKendra I made a reproduction repo here; if anyone has issues reproducing let me know and I can help explain!

[andreialecu]

Looks like we got hit by this too. The stack is now in UPDATE_ROLLBACK_FAILED state and cannot seemed to be rolled back.

Any workarounds to get it un stuck? Would skipping the resource be safe?

[candu]

I can see the challenge here: if I export a cross-region reference from stack A to stack B, and stack B depends on resources in stack A, and I delete the resource this cross-region reference refers to, I then have a cyclic update dependency.

From the ADR linked above, I'd want to update stack B (export reader), then stack A (export writer).

However, since stack B depends on resources in stack A, I'd want to update stack A, then stack B.

As a result, neither stack is safe to update before the other.

Maybe an ideal resolution would be to detect the cyclic dependency here and fast-fail? In cases like this, I'd much rather have fast-fail behaviour than unblock UPDATE_ROLLBACK_FAILED snags (e.g. with a warning that guides me to use --exclusively, or to break up the update into two steps).

[AposLaz]

Any updates for this issue?

[ruslan-roboto]

This is giving us headaches as well. Any update would be immensely helpful. Thanks!

[flochaz]

On my side, got it fixed by:

1. Reverted manually to old values in SSM (check your CustomCrossRegionExportWriter Lambda logs in cloudwatch to get the old and new value and be careful in the logs between the one that corespond to the initial rewrite and the one triggered by the rollback :) ) WITHOUT THE aws-cdk:strong-ref TAG !
2. Trigger a "Continue update rollback" of the cloudformation stack in UPDATE_ROLLBACK_FAILED without the failing CustomCrossRegionExportWriter resource (check the box in the advanced continue rollback modal)
3. Redeploy the stacks

[JonWallsten]

I see two EASY solutions conceptually, but maybe harder technically. Both would require to allow a way to keep the reference in the ExportWriter resource, but remove it from the ExportReader. a) Either by adding it manually to the ExportWriter in the producing stack b) Have some kind of DummyImport in the receiving stack that only creates the export and not the import

Then you can really easily create a new resource along the old if you need to replace something, like a Certificate, or remove the resource if that's what you need.

[renschler]

Coming back to this issue because I hit it again. I'm just going to leave my notes here; they aren't that great but maybe someone will find it helpful.

I have a stack (named "A") that has a lambda function which triggers statemachines in other stacks ("A-region1", "A-region2", "A-region3", "A-region4" etc..); those stacks are cross-region.

There's currently an issue where if your crossRegionReferences parameters are too long the deploy fails https://github.com/aws/aws-cdk/issues/30119 .

I had added too many regions, so I ran into that issue and the deploy failed.

To fix things I thought I could just remove the additional regions (ex: remove A-region3 and A-region4) from my cdk_resources.ts file and try the deploy again.

This attempted patch-deploy then also failed. In particular I had a stack B which failed with the error Exports cannot be updated and it got stuck in the UPDATE_ROLLBACK_FAILED state.

Stack B was listed as a dependency for all the statemachine stacks (A-region1, A-region2, A-region3, A-region4).

I think what was happening is that this followup deploy was trying to change the Stack B exports to no longer pass to region3 and region4, but since at that point the stacks A-region3 and A-region4 hadn't been deleted yet, you can't proceed because those exports are currently in use.

What I did....

From the aws web console I restarted the rollback update for stack B and checked the advanced boxes to skip rolling back the failing resources for Stack B. This completed successfully.

Then I made sure my A-region3 and A-region4 were still in my cdk_resources, and from the terminal I ran cdk destroy A-region3 A-region4. Those were successfully destroyed, then I removed them from the cdk-resources ,and redeployed Stack B and it was no longer in the update rollback failed state.

I've simplified my exact scenario a bit here. In reality my A-region stacks were listed as dependencies for another stack C, so things were messier... I had to first make sure that stack C was no longer referencing the region3/region4 regions that were going to be destroyed, only then could i proceed with the cdk destroy command because I didn't want to destroy stack C.

[renschler]

Found some more references to this issue: https://x.com/theburningmonk/status/1826268576187060613 https://sst.dev/blog/moving-away-from-cdk/#3-export-in-use

[horyd]

I managed to fix my cross region issues, finally. For context, I had a certificate in us-east-1 that I wanted to make changes to (or more realistically, recreate and remove the old one). The certificate was used in a cloudfront distribution in my second stack in eu-west-2.

1. Create a new certificate in stack 1. In stack 2, reference the new certificate in the CF distribution. Also in stack 2, add a CFNOutput for the original certificate (in my case I output the original certificate ARN). This second step is critical, as it makes sure the ExportsWriter does not get removed in stack 1 (which is what causes failures).
2. Deploy both stacks.
3. Remove the CFNOutput from stack 2. EXCLUSIVELY deploy stack 2 using --exclusively. This will remove the reader.
4. Deploy stack 1. It will see the reference to the old certificate is no longer needed and remove the writer.
5. Remove the certificate and deploy stack 1 again.