adopt-or-create adoption policy for ResourceAdoption alpha feature

[havard024]

Creating this issue to start an open discussion on UX for the adopt-or-create adoption policy for ResourceAdoption alpha feature.

Link to ResourceAdoption alpha feature current documentation https://aws-controllers-k8s.github.io/community/docs/user-docs/features/#resourceadoption

Reference to slack discussion regarding this issue https://kubernetes.slack.com/archives/C0402D8JJS1/p1740081894642539

Description of our use case / possible user experience:

For some aws resources that either take a long time to create, eg cloudfront distributions or other aws resources where we want to retain data, like s3 buckets or rds, we would like to keep the created resources even if the manifests defining them are deleted in kubernetes.

Use cases

• when developing kro stacks with ack

when I develop stacks using kro and ack, I often delete / recreate the stack, particularly after I think I'm done and want to see how kro behaves when there are no resources in place. If this stack contains a cloudfront distribution and I currently work on resources that depend on that distribution, if I were to delete the stack, I need to wait for the cloudfront distribution to be created again which slows down my development cycle.

That said, I might not need adopt-or-create for this use case, if I instead could delete the resources I want to create manually from k8s and trigger a kro reconciliation of the stack.

• disaster recovery / accidental deletion of k8s manifests / gitops reconciliation

to be able to reuse the manifests used to create the aws resources that has been retained

[github-actions[bot]]

Hello @havard024 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[michaelhtm]

Hello @havard024, It seems like you're describing the deletion-policy feature which we do support in ACK. Here's the documentation. https://aws-controllers-k8s.github.io/community/docs/user-docs/deletion-policy/

adopt-or-create feature is meant to attempt adopting a resource from AWS if it exists, if it can't find it, it tries creating it. Is that what you're looking for?

[havard024]

Hi @michaelhtm, adopt-or-create is exactly what I'm looking for.

If I have a resources with deletion-policy on and the ack manifest that created that resource disappears for any reason, I don't want to adopt the resource if it already exists instead of recreating.

accidental deletes of ack manifests using gitops where the corresponding ack resource has deletion-policy=retain

disaster recovery, the kubernetes cluster is gone, but not necessarily the aws resources, we want to adopt them instead of creating them using the same ack manifest.

[michaelhtm]

If I have a resources with deletion-policy on and the ack manifest that created that resource disappears for any reason, I don't want to adopt the resource if it already exists instead of recreating.

can you elaborate more on this? i'm not sure i understand...

[gfaraj]

Hi! I have a similar use case here:

• We have many existing AWS resources that have been created by Terraform in the past
• When deploying an application to an environment, we don't know if the AWS resource exists or not yet (it could be a new environment recently set up)
• Ideally the K8s ACK CR would define the new desired state of either an existing AWS resource, or a new AWS resource.

Right now I'm getting this error:

QueueAlreadyExists: A queue already exists with the same name and a different value for tags

Hoping this gets traction soon!

[JonathanGraniero]

Hello! Similar to what @gfaraj mentioned, I'm working on a project currently that is in the same boat, we have a lot of resources that are created in terraform, and we are looking to do the migration to using ACK.

The issue we are running into is that we want to define one resource definition, and have it deployable across multiple deployments. This means that right now we have to have complex logic or specific workflows around adopting a resource, then back porting the adopted resource. We would love to have a mechanism to add an annotation for adopt-or-create so that way it'll automatically create the resource if it doesn't exist, and for existing deployments it doesn't create any new resources and we still have visibility into maintaining those resources.

[havard024]

If I have a resources with deletion-policy on and the ack manifest that created that resource disappears for any reason, I don't want to adopt the resource if it already exists instead of recreating.

can you elaborate more on this? i'm not sure i understand...

@michaelhtm sorry about that, it basically as @gfaraj mentioned, if you have an existing aws resources with no ack manifest defined (eg you are migrating from something else to ack) or if you have an ack manifest with deletion-policy set to retain and for some reason lose the ack manifest (eg accidental delete), I can't add the ack manifest back because it will try to create a new resource.

With a adopt-or-create policy, we expect ack to take control over the state when adopting the resource back.

[michaelhtm]

Hello, We have an initial implementation in the works. Will keep you all updated on here!

[gfaraj]

Hi @michaelhtm -- looks like this change was merged, nice! Is this planned to be released soon?

[michaelhtm]

Hi @gfaraj Yes, we are starting the release process soon!

[michaelhtm]

Hello @gfaraj @havard024 @JonathanGraniero,

adopt-or-create adoption policy is now supported in all controllers! Here's a link to the updated documentation: https://aws-controllers-k8s.github.io/community/docs/user-docs/features/#resourceadoption

[gfaraj]

@michaelhtm can I assume it's in all the helm charts released 4 days ago? Like sqs-chart 1.1.10? Thanks for pushing this out!

[gfaraj]

I'm not sure I completely grasp this sentence in the docs:

If the read operation required field is in the status the adoption-fields annotation will be used to retrieve such fields.

Is the adoption-fields annotation optional for adopt-or-create? If, for example for a Queue, I have queueName in the spec, is that enough for the read operation and I can omit the annotation?

[michaelhtm]

Yes that is correct. In the EKS Cluster example in the docs, you can see that we don't have adoption-fields since we already defined spec.name. Whereas if it is a VPC, we would need to define the vpcID in the annotation.

There was a small bug when using adopt-or-create for resources that require adoption-fields. We will release the fix by the end of the week!

[gfaraj]

Can you describe the bug? Does that impact SQS? I'm seeing issues and I'm not sure if that's related to the bug you're referring to or something I did wrong.

[gfaraj]

The issue that I'm seeing is that adding the adopt-or-create adoption-policy to an existing ACK Queue does not seem to take effect -- still getting the QueueAlreadyExists error. Deleting it and creating a new one works perfectly though. Is this expected?

[havard024]

@michaelhtm

Hello @gfaraj @havard024 @JonathanGraniero,

adopt-or-create adoption policy is now supported in all controllers! Here's a link to the updated documentation: https://aws-controllers-k8s.github.io/community/docs/user-docs/features/#resourceadoption

awesome! 💯

I've managed to use it successfully on a bucket, certificate, policy.

I'm having issues when I try to use it on a OriginAccessControl. I need to specify the id in the adoption fields to make it work

services.k8s.aws/adoption-fields: |
      {
        "id": "..."
      }

When successfully adopting the OAC using the id in adoption-fields I noticed the log output of the cloudfront controller seems to claim that the resource was not adopted "is_adopted":false

{
    "level": "info",
    "ts": "2025-05-07T04:38:43.267Z",
    "logger": "ackrt",
    "msg": "desired resource state has changed",
    "kind": "OriginAccessControl",
    "namespace": "***",
    "name": "***",
    "account": "***",
    "role": "arn:aws:iam::***:role/***",
    "region": "***",
    "is_adopted": false,
    "generation": 1,
    "diff": [
        {
            "Path": {
                "Parts": [
                    "Spec",
                    "OriginAccessControlConfig",
                    "Description"
                ]
            },
            "A": "",
            "B": null
        }
    ]
}
{
    "level": "info",
    "ts": "2025-05-07T04:38:43.462Z",
    "logger": "ackrt",
    "msg": "updated resource",
    "kind": "OriginAccessControl",
    "namespace": "***",
    "name": "***",
    "account": "***",
    "role": "arn:aws:iam::***:role/***",
    "region": "***",
    "is_adopted": false,
    "generation": 1
}

It looks like the status.id field of the OriginAccessControl is not populated when adopting it, I reference the field in my cloudfront distribution.

status fields of adopted OAC

  status:
    ackResourceMetadata:
      ownerAccountID: "***"
      region: ***
    conditions:
    - lastTransitionTime: "2025-05-07T04:58:13Z"
      message: Resource synced successfully
      reason: ""
      status: "True"
      type: ACK.ResourceSynced
    eTag: E3LJ5WMKNRFKQS

I'm also having issues adopting a cloudfront distribution, I've tried adding id to the adoption-fields without any luck, here's the status of the distribution when trying to adopt an existing

status:
    ackResourceMetadata:
      ownerAccountID: "***"
      region: ***
    conditions:
    - lastTransitionTime: "2025-05-07T05:32:18Z"
      status: "True"
      type: ACK.ReferencesResolved
    - message: 'CNAMEAlreadyExists: One or more of the CNAMEs you provided are already
        associated with a different resource.'
      status: "True"
      type: ACK.Recoverable
    - lastTransitionTime: "2025-05-07T05:32:19Z"
      message: Unable to determine if desired resource state matches latest observed
        state
      reason: 'operation error CloudFront: CreateDistribution, https response error
        StatusCode: 409, RequestID: 1bd541c2-9af8-4f72-b9ea-7d4eae8dfd6c, CNAMEAlreadyExists:
        One or more of the CNAMEs you provided are already associated with a different
        resource.'
      status: Unknown
      type: ACK.ResourceSynced

[michaelhtm]

Hello @gfaraj The issue is, when you have adopt-or-create policy, and you need to define adoption-fields (the field you need to populate in the status), that field is not getting patched to the resource. Here's a similar issue: https://github.com/aws-controllers-k8s/community/issues/2459

Yes, SQS would also be affected by this. There's a fix for it that will be released in all controllers soon!

[michaelhtm]

@havard024 Your issue is also part of the same bug. Release incoming! Including the is_adopted log fix!

[gfaraj]

@michaelhtm just to totally confirm this is the same bug, my issue happens when:

1. Create an ACK Queue resource that does not have an adoption policy and points to an existing queue.
2. This causes ACK to fail to reconcile with QueueAlreadyExists.
3. Update the existing ACK Queue resource to have adopt-or-create adoption policy and adoption-fields with queueURL pointing to the existing queue.
4. ACK fails to reconcile still with QueueAlreadyExists.

But this works:

1. Create an ACK Queue resource with adopt-or-create adoption policy and adoption-fields with queueURL pointing to the existing queue.
2. ACK reconciles and adopts the resource correctly.

[michaelhtm]

Seems like the controller is managing (setting finalizer) before the resource is successfully created. https://github.com/aws-controllers-k8s/runtime/blob/7c4820749466d03a4a7d54b2af6896606b734466/pkg/runtime/reconciler.go#L618-L633

Adoption is triggered only when resource is not managed yet.

@a-hilaly Is there a reason why we manage resources before they are created successfully?

[gfaraj]

@michaelhtm @a-hilaly any update on the above question?

[michaelhtm]

Hello @gfaraj So the reason marking a resource as managed (putting the finalizer) before attempting a create is a general practice in kubernetes. The main reason we do it is to protect against deletion protection. If we don't put the finalizer, there is no deletion protection against the resource. Eg: If you create a resource, and delete it immediately, it would be deleted from your cluster, because there's no finalizer, but remain in AWS.

The current adoption logic expects the resource to not be managed (no finalizer) to trigger an adoption. The issue you encountered is specific to few controllers, where you need a QueueURL for SQS Queue, or an ARN for IAM Policy to adopt, but the name of the resource must also be unique. Since you already try to create the resource, the controller will mark it as managed, before attempting a create. When you later try to add the adoption-policy and adoption-fields, the controller ignores them since the resource is already managed (as seen here).

The best approach to resolve this now would be to mark a resource as unmanaged whenever you get a any AWS exception. This would allow you to decide adopting a resource.

[michaelhtm]

Also @havard024, We just made releases in all controllers with a fix to your issue! Feel free to drop comments and feedback :)

[havard024]

Also @havard024, We just made releases in all controllers with a fix to your issue! Feel free to drop comments and feedback :)

@michaelhtm comments below:

It looks like the status.id field of the OriginAccessControl is not populated when adopting it, I reference the field in my cloudfront distribution.

This works, the status.id field is populated when adopting 👍

I'm having issues when I try to use it on a OriginAccessControl. I need to specify the id in the adoption fields to make it work

I still need to specify the OriginAccessControl.id in the adption fields for the OriginAccessControl resource for it to be adopted.

I'm also having issues adopting a cloudfront distribution, I've tried adding id to the adoption-fields without any luck, here's the status of the distribution when trying to adopt an existing

I'm able to adopt this as long as I specify the distribution id in adoption-fields. Would love it if I did not need to specify adoption fields.

---

All in all, adopting works and the status field is populated. 💯

It would be create if I could adopt without specifying adoption fields for the following resources:

• origin access control
• distribution
• policy <- this I could live without since I have deterministic policy name and can construct the arn based on the available parameters

The gitops flow breaks since I have to manually specify the adoption fields after creating the resources the first time.

[michaelhtm]

@havard024

This works, the status.id field is populated when adopting 👍

That's great to hear

I still need to specify the OriginAccessControl.id in the adption fields for the OriginAccessControl resource for it to be adopted.

adoption-fields allows you to define fields that need to go in the status. With the current implementation, if you don't provide one, the controller would just create a new resource.

Also, for Resources like Cloudfront Distribution, the only unique thing about them is their ID. which goes in the Status, and if you want to adopt a specific Distribution, you would need to provide one.

[michaelhtm]

@gfaraj

Create an ACK Queue resource that does not have an adoption policy and points to an existing queue. This causes ACK to fail to reconcile with QueueAlreadyExists. Update the existing ACK Queue resource to have adopt-or-create adoption policy and adoption-fields with queueURL pointing to the existing queue. ACK fails to reconcile still with QueueAlreadyExists. But this works:

Create an ACK Queue resource with adopt-or-create adoption policy and adoption-fields with queueURL pointing to the existing queue. ACK reconciles and adopts the resource correctly.

This fix has now been released in all controllers. Closing this issue for now. Please feel free to open a new one, if there are any more issues. Thanks :) /close

[ack-prow[bot]]

@michaelhtm: Closing this issue.

In response to this:

@gfaraj

Create an ACK Queue resource that does not have an adoption policy and points to an existing queue. This causes ACK to fail to reconcile with QueueAlreadyExists. Update the existing ACK Queue resource to have adopt-or-create adoption policy and adoption-fields with queueURL pointing to the existing queue. ACK fails to reconcile still with QueueAlreadyExists. But this works:

Create an ACK Queue resource with adopt-or-create adoption policy and adoption-fields with queueURL pointing to the existing queue. ACK reconciles and adopts the resource correctly.

This fix has now been released in all controllers. Closing this issue for now. Please feel free to open a new one, if there are any more issues. Thanks :) /close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.