aws-cdk icon indicating copy to clipboard operation
aws-cdk copied to clipboard

(aws-eks-v2-alpha): kubectl-based resources may not be successfully created/deleted

Open msessa opened this issue 5 months ago • 1 comments

Describe the bug

When creating/deleting resources that rely on the kubectl provider (eg. HelmChart, KubernetesManifest) it is possible that the AccessEntry for the kubectl provider could not yet have been created or, in case of a stack deletion, have been deleted before those resources, leaving kubectl without the appropriate permissions.

Regression Issue

  • [ ] Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

The AccessEntry for the kubectl provider should be included in the KubectlReadyBarrier's dependencies to ensure correct ordering when creating/deleting resources

Current Behavior

The AccessEntry for the kubectl is not explicitly marked as a dependency of the KubectlReadyBarrier resource. Opening the possibility of incorrect order of operation especially during delete.

As an example, these are the CR handler logs of a HelmChart being deleted, which happens after the AccessEntry has already been deleted:

[INFO]	2025-07-03T02:32:51.662Z	f0b1956e-0a0b-4a00-b917-020cff7feb80	{
    "RequestType": "Delete",
    "ServiceToken": "...",
    "ResponseURL": "...",
    "StackId": "...",
    "RequestId": "3585eea3-9049-4f93-9f87-287315d7e681",
    "LogicalResourceId": "SolrZooKeeperChart41F0FD45",
    "PhysicalResourceId": "...",
    "ResourceType": "Custom::AWSCDK-EKS-HelmChart",
    "ResourceProperties": {
        "ServiceToken": "...",
        "Values": "...",
        "ClusterName": "SolrCluster46192153-kKOPdVgcdKuy",
        "Wait": "true",
        "Release": "zookeeper",
        "Chart": "oci://registry-1.docker.io/bitnamicharts/zookeeper",
        "Namespace": "default"
    }
}
Updated context arn:aws:eks:ap-southeast-2:...:cluster/SolrCluster46192153-kKOPdVgcdKuy in /tmp/kubeconfig
[INFO]	2025-07-03T02:32:54.747Z	f0b1956e-0a0b-4a00-b917-020cff7feb80	delete error: b'Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials\n'

Reproduction Steps

  • Create a cluster with kubectl support, and at least one HelmChart or KubernetesManifest resource
  • Delete the stack

observe the AccessEntry for kubectl provider being immediately deleted. observe errors in deleting the HelmChart or KubernetesManifest within the logs of the kubectl handler

Possible Solution

No response

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.203.0

AWS CDK CLI version

latest

Node.js Version

20

OS

Mac

Language

TypeScript

Language Version

No response

Other information

No response

msessa avatar Jul 03 '25 03:07 msessa

Hi @msessa,

Thank you for reporting this issue and for already providing a comprehensive fix in PR #34898!

Issue Confirmed: You've correctly identified a dependency ordering problem in the EKS v2 alpha implementation. The AccessEntry for the kubectl provider was not being added as a dependency to the KubectlReadyBarrier, causing authentication failures during stack deletion.

Your Fix is Spot On: The solution in PR #34898 correctly adds the missing dependency:

this._kubectlReadyBarrier.node.addDependency(this._clusterAdminAccess);

This ensures that kubectl-based resources are always deleted before their required AccessEntry, preventing the "server has asked for client to provide credentials" errors you observed.

Impact: This fix will significantly improve the reliability of EKS v2 alpha cluster teardown operations, especially for clusters using HelmChart and KubernetesManifest resources.

The PR looks well-structured with appropriate test updates. Thank you for both identifying the issue and providing a clean, targeted solution!

pahud avatar Jul 03 '25 15:07 pahud