terraform-provider-kubectl
terraform-provider-kubectl copied to clipboard
gavinbunney
<name> failed to fetch resource from kubernetes: the server could not find the requested resource
The issue
I'm running into an error with karpenter yaml templates and I'm unsure on what the cause is. I've used the kubectl_manifest in the past and it worked fine on consecutive applies, but for some reason it's not working with these custom resources.
EKS version: 1.27 gavinbunney kubectl version: 1.14 terraform: 1.4.6
Sample YAML file:
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: system-infra
spec:
instanceProfile: my_cluster-worker-role
securityGroupSelector:
kubernetes.io/cluster/my_cluster: owned
subnetSelector:
karpenter.sh/discovery: eks-private
tags:
Name: node
managedBy: Terraform
project: Sandbox
repository: ..........
stack: stack/sandbox
workspace: default
When I try to apply it through terraform it will plan just fine, but when I try to apply it, it gives me the following error:
╷
│ Error: permissionsets failed to fetch resource from kubernetes: the server could not find the requested resource
│
│ with module.core_system.kubectl_manifest.permissions-sets[0],
│ on ......./rbac-addon.tf line 123, in resource "kubectl_manifest" "permissions-sets":
│ 123: resource "kubectl_manifest" "permissions-sets" {
│
╵
I don't know where to start looking to resolve this. The items never show up in the statefile, but the objects are created inside the cluster. It looks like it created them and tries to find them, but then doesn't seem to find them?
I'll keep digging as this provider is my only way of applying yamls to the cluster through terraform without using null_resource ( dirty last resort hack imo ) or the kubernetes_manifest ( which doesn't work if the CRDs don't exist ).
I'm not sure if this is related to the fact these are custom resources ( through CRDs ).
First thought - could this be related to the fact that the resource is not a namespaced resource, but rather a cluster resource ( like a cluster role )?
I'm testing with lower kubernetes versions. Possibly related to the helm charts being incompatible with those versions.
Problem appears resolved on EKS v1.26. If other people run into this error. This might be your solution. I'll leave the ticket as I'm unsure if this issue is related to the charts being incompatible ( which I'm guessing ), or if it's related to the kubectl provider. I'll leave it up to the maintainer.
i am seeing same issue on EKS kubernetes 1.27 with karpenter 0.27.5 and kubectl provider 1.14.0
I' am also seeing same behavior. EKS 1.27, Kubectl provider 1.14.0. I'm trying just apply ENIconfig after EKS creation with multiple subnets.
Downgrade to EKS 1.26 resolved it.
My EKS 1.27 also has the issue related to the provider and karpenter:
│ Error: default failed to fetch resource from kubernetes: the server could not find the requested resource
│
│ with module.karpenter.kubectl_manifest.karpenter_provisioner,
│ on ../../../../tf_modules/terraform-aws-eks-karpenter/main.tf line 69, in resource "kubectl_manifest" "karpenter_provisioner":
│ 69: resource "kubectl_manifest" "karpenter_provisioner" {
I found in the EKS logs that a request was made to the Kubernetes API with an incorrect path, resulting in a 404 error response. It appears that the provider is making the request with an extra ApiVersion, causing it to be duplicated in the path.
"requestURI": "/apis/karpenter.k8s.aws/v1alpha1/v1alpha1/awsnodetemplates/default",
Full log entry:
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"stage": "ResponseComplete",
"requestURI": "/apis/karpenter.k8s.aws/v1alpha1/v1alpha1/awsnodetemplates/default",
"verb": "get",
"user": {
"username": "kubernetes-admin",
"groups": [
"system:masters",
"system:authenticated"
],
"extra": {
...
]
}
},
"sourceIPs": [
],
"userAgent": "HashiCorp/1.0 Terraform/1.5.0",
"objectRef": {
"resource": "v1alpha1",
"name": "awsnodetemplates",
"apiGroup": "karpenter.k8s.aws",
"apiVersion": "v1alpha1",
"subresource": "default"
},
"responseStatus": {
"metadata": {},
"code": 404
},
"requestReceivedTimestamp": "2023-06-14T12:07:55.982587Z",
"stageTimestamp": "2023-06-14T12:07:55.982853Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": ""
}
}
Also getting this in EKS 1.27, but in my case, I'm creating a ClusterSecretStore for external-secrets. Edit, gonna try helm_release with the https://charts.itscontained.io/ "raw" chart... if it works I'll report back.
OK, it worked. However, I had to download the chart and store it with my module because it's no longer hosted above. But it's a temporary workaround for my issue - the raw chart applies my ClusterSecretStore.
Same here, happens with multiple resources - node template and provisioner for karpenter or horizontalrunnerautoscaler for actions-runner-controller
$ k version --short
Client Version: v1.27.1
Kustomize Version: v5.0.1
Server Version: v1.27.2-eks-c12679a
Seems to be a generic issue for 1.27 - @gavinbunney any change you could look into this? Downgrading is not an options for many clusters...
More debugging ensued, switching to the kubernetes_manifest resource kinda figured there is a delta between what is being applied from the yaml and what the kube control plane returns the object as. For example
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.karpenter_provisioner_amd64, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.spec.requirements: new element 6 has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.karpenter_provisioner_arm64, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.spec.requirements: new element 6 has appeared.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
Checking the resource yaml on the cluster and reconciling it back to the code fixed the issue on the kubernetes_manifest provider.
Another example
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.runners-amd64, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.spec.template.spec.resources.requests["cpu"]: was cty.StringVal("1.6"), but now cty.StringVal("1600m").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.runners-amd64, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.spec.template.spec.resources.limits["cpu"]: was cty.StringVal("4.0"), but now cty.StringVal("4").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
Suggest trying the same with your resources and analyzing, or you could switch to the kubernetes_manifest resource.
I have same issue trying to apply EniConfig resource on EKS cluster version 1.27. It worked on first apply but failed after that.
Since everyone in this issue seems to be talking about EKS, I'd like to add that we've run into the same issue with kubeadm based clusters on GCP. So it's definitely a 1.27 thing and not platform specific.
I'll also add that I've started seeing this since upgrading a K3S cluster to 1.27.3+k3s1, so that's one more point in the "this is a 1.27 issue" column!
@seungmun per @tejisin's comment I switched all my kubectl_manifest resources to kubernetes_manifest which works fine. Some of my resources needed to be redeployed IIRC, but in my case that wasn't an issue.
@karmajunkie Thanks for the super fast feedback. Are you sure you meant to use the resources in hashicorp/terraform-provider-kubernetes?
@seungmun yep, that's the one. I'm not sure if that's going to be universally true but my needs are pretty straightforward.
The Problem with hashicorp/kubernetes is that it needs CRDs to be present in the plan phase if you are creating custom resources, so you'd need to have 2 separate Terraform configurations with 2 separate states if that's the case. See https://github.com/hashicorp/terraform-provider-kubernetes/issues/1367
The other option is to use a null_resource or similar with a provisioner block that runs kubectl to apply manifests as a workaround until this issue is fixed.
I believe you also can't create e.g. the EKS cluster from a scratch and apply your manifests in the same terraform step with kubernetes_manifest - that was the initial reason I switched to this kubectl_manifest.
Let's be realistic though - this project has been abandoned. It would be awesome if @gavinbunney had time once more to update the code or add some other guys as maintainers but that might be unlikely.
For me it's the mentioned problem with CRDs and CRs. I'm fairly sure the problem is related to the versions of the libraries used (prime suspect being the ancient Kubernetes client libraries) but I've never built a Terraform provider.
I've tried to create a fork and update the go modules. While this worked eventually to build some binaries after some minor code changes, the subsequent terraform apply unfortunately errored with issues I also get from the hashicorp kubectl provider (like "kind" unknown, "group" issues, etc.).
I am not versed enough to continue from here onwards but it might be that a simple update of the underlying modules/libraries is not enough to reinstate the previous behavior of this provider here, given all the upstream changes in the k8s API.
I'm experiencing similar issues with the kubectl provider on EKS cluster v1.27. Sometimes provider just drops resources from the tf state because it can not find them. When I try to import resources back with tf import ... I get an error: failed to fetch resource from kubernetes: ....
Here are debug logs from tf plan during which resources are removed from the state (just interesting parts):
2023-07-04T18:42:27.207+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [WARN] kubernetes resource (/apis/crd.k8s.amazonaws.com/v1alpha1/eniconfigs/eu-central-1b) not found, removing from state
2023-07-04T18:42:27.207+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [WARN] kubernetes resource (/apis/crd.k8s.amazonaws.com/v1alpha1/eniconfigs/eu-central-1c) not found, removing from state
2023-07-04T18:42:27.207+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [WARN] kubernetes resource (/apis/crd.k8s.amazonaws.com/v1alpha1/eniconfigs/eu-central-1a) not found, removing from state
2023-07-04T18:42:27.207+0200 [WARN] Provider "registry.terraform.io/gavinbunney/kubectl" produced an unexpected new value for module.eks_euc1.kubectl_manifest.eni_config["eu-central-1a"] during refresh.
2023-07-04T18:42:27.207+0200 [WARN] Provider "registry.terraform.io/gavinbunney/kubectl" produced an unexpected new value for module.eks_euc1.kubectl_manifest.eni_config["eu-central-1c"] during refresh.
2023-07-04T18:42:27.207+0200 [WARN] Provider "registry.terraform.io/gavinbunney/kubectl" produced an unexpected new value for module.eks_euc1.kubectl_manifest.eni_config["eu-central-1b"] during refresh.
2023-07-04T18:42:27.209+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [DEBUG] eu-central-1b Unstructed YAML: map[apiVersion:crd.k8s.amazonaws.com/v1alpha1 kind:ENIConfig metadata:map[name:eu-central-1b] spec:map[securityGroups:[sg-0329cc8efeae8b0cf sg-0efb5545ec1b733ef] subnet:subnet-073fc7974ff152aa8]]
2023-07-04T18:42:27.209+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [DEBUG] eu-central-1c Unstructed YAML: map[apiVersion:crd.k8s.amazonaws.com/v1alpha1 kind:ENIConfig metadata:map[name:eu-central-1c] spec:map[securityGroups:[sg-0329cc8efeae8b0cf sg-0efb5545ec1b733ef] subnet:subnet-0200bdef6e567c74f]]
2023-07-04T18:42:27.209+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [DEBUG] eu-central-1b Unstructed YAML: map[apiVersion:crd.k8s.amazonaws.com/v1alpha1 kind:ENIConfig metadata:map[name:eu-central-1b] spec:map[securityGroups:[sg-0329cc8efeae8b0cf sg-0efb5545ec1b733ef] subnet:subnet-073fc7974ff152aa8]]
2023-07-04T18:42:27.209+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [DEBUG] eu-central-1c Unstructed YAML: map[apiVersion:crd.k8s.amazonaws.com/v1alpha1 kind:ENIConfig metadata:map[name:eu-central-1c] spec:map[securityGroups:[sg-0329cc8efeae8b0cf sg-0efb5545ec1b733ef] subnet:subnet-0200bdef6e567c74f]]
2023-07-04T18:42:27.209+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [DEBUG] eu-central-1a Unstructed YAML: map[apiVersion:crd.k8s.amazonaws.com/v1alpha1 kind:ENIConfig metadata:map[name:eu-central-1a] spec:map[securityGroups:[sg-0329cc8efeae8b0cf sg-0efb5545ec1b733ef] subnet:subnet-0c3c1b5745dbdad8c]]
2023-07-04T18:42:27.209+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2023/07/04 18:42:27 [DEBUG] eu-central-1a Unstructed YAML: map[apiVersion:crd.k8s.amazonaws.com/v1alpha1 kind:ENIConfig metadata:map[name:eu-central-1a] spec:map[securityGroups:[sg-0329cc8efeae8b0cf sg-0efb5545ec1b733ef] subnet:subnet-0c3c1b5745dbdad8c]]
2023-07-04T18:42:27.210+0200 [WARN] Provider "registry.terraform.io/gavinbunney/kubectl" produced an invalid plan for module.eks_euc1.kubectl_manifest.eni_config["eu-central-1c"], but we are tolerating it because it is using the legacy plugin SDK.
2023-07-04T18:42:27.210+0200 [WARN] Provider "registry.terraform.io/gavinbunney/kubectl" produced an invalid plan for module.eks_euc1.kubectl_manifest.eni_config["eu-central-1a"], but we are tolerating it because it is using the legacy plugin SDK.
2023-07-04T18:42:27.210+0200 [WARN] Provider "registry.terraform.io/gavinbunney/kubectl" produced an invalid plan for module.eks_euc1.kubectl_manifest.eni_config["eu-central-1b"], but we are tolerating it because it is using the legacy plugin SDK.
2023-07-04T18:42:27.211+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/gavinbunney/kubectl/1.14.0/darwin_arm64/terraform-provider-kubectl_v1.14.0 pid=16928
2023-07-04T18:42:27.834+0200 [DEBUG] Resource state not found for node "module.eks_euc1.kubectl_manifest.eni_config[\"eu-central-1c\"]", instance module.eks_euc1.kubectl_manifest.eni_config["eu-central-1c"]
2023-07-04T18:42:27.834+0200 [DEBUG] Resource state not found for node "module.eks_euc1.kubectl_manifest.eni_config[\"eu-central-1a\"]", instance module.eks_euc1.kubectl_manifest.eni_config["eu-central-1a"]
2023-07-04T18:42:27.834+0200 [DEBUG] Resource state not found for node "module.eks_euc1.kubectl_manifest.eni_config[\"eu-central-1b\"]", instance module.eks_euc1.kubectl_manifest.eni_config["eu-central-1b"]
I've managed to find a workaround that works for my use case. I've switched to helm_release resource from the official hashicorp/helm provider. And I'm using dysnix/raw chart which is basically an empty chart that takes raw K8s yaml resources as input. Original comment which suggests this approach: https://github.com/hashicorp/terraform-provider-kubernetes/issues/1380#issuecomment-962058148
I've tested this by creating a new cluster from scratch. Old:
resource "kubectl_manifest" "eni_config" {
for_each = zipmap(var.azs, var.pods_subnet_ids)
yaml_body = yamlencode({
apiVersion = "crd.k8s.amazonaws.com/v1alpha1"
kind = "ENIConfig"
metadata = {
name = each.key
}
spec = {
securityGroups = [
module.eks.cluster_primary_security_group_id,
module.eks.node_security_group_id,
]
subnet = each.value
}
})
}
New:
resource "helm_release" "eni_configs" {
name = "eni-configs"
repository = "https://dysnix.github.io/charts"
chart = "raw"
version = "v0.3.2"
values = [
yamlencode({
resources = [
for az, subnet_id in zipmap(var.azs, var.pods_subnet_ids) :
{
apiVersion = "crd.k8s.amazonaws.com/v1alpha1"
kind = "ENIConfig"
metadata = {
name = az
}
spec = {
securityGroups = [
module.eks.cluster_primary_security_group_id,
module.eks.node_security_group_id,
]
subnet = subnet_id
}
}
]
})
]
}
For anybody stuck, blocked, losing hope for a progress on that - if you are in need of deploying your CRs before CRDs are available then I would suggest to do what I've done for now, eg. before:
resource "kubectl_manifest" "envoyfilter-proxy_protocol-internal" {
yaml_body = <<-EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: proxy-protocol-internal
namespace: istio-system
spec:
configPatches:
- applyTo: LISTENER
patch:
operation: MERGE
value:
listener_filters:
- name: envoy.filters.listener.proxy_protocol
- name: envoy.filters.listener.tls_inspector
workloadSelector:
labels:
istio: ingressgateway-internal
EOF
depends_on = [
helm_release.istio-istiod
]
}
after:
resource "helm_release" "envoyfilter-proxy-protocol-internal" {
name = "envoyfilter-proxy-protocol-internal"
namespace = kubernetes_namespace.istio-system.metadata[0].name
repository = "https://bedag.github.io/helm-charts/"
chart = "raw"
version = "2.0.0"
values = [
<<-EOF
resources:
- apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: proxy-protocol-internal
namespace: istio-system
spec:
configPatches:
- applyTo: LISTENER
patch:
operation: MERGE
value:
listener_filters:
- name: envoy.filters.listener.proxy_protocol
- name: envoy.filters.listener.tls_inspector
workloadSelector:
labels:
istio: ingressgateway-internal
EOF
]
depends_on = [
helm_release.istio-istiod,
kubernetes_namespace.istio-ingress
]
}
A dirty workaround that works
locals {
karpenter_aws_node_template_yaml = <<EOF
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: ${local.karpenter_provisioner_name}
spec:
blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
deleteOnTermination: ${var.karpenter_provisioner_ebs_delete_on_termination}
volumeSize: ${var.karpenter_provisioner_ebs_size}
volumeType: ${var.karpenter_provisioner_ebs_type}
subnetSelector:
karpenter.sh/discovery: ${module.eks.cluster_name}
securityGroupSelector:
karpenter.sh/discovery: ${module.eks.cluster_name}
tags:
karpenter.sh/discovery: ${module.eks.cluster_name}
EOF
}
resource "null_resource" "install_karpenter_aws_node_template" {
triggers = {
timestamp = timestamp()
}
provisioner "local-exec" {
interpreter = ["/bin/bash", "-c"]
command = "aws eks --region ${var.region} update-kubeconfig --name ${var.cluster_name} && kubectl apply -f - <<EOF\n${local.karpenter_aws_node_template_yaml}\nEOF"
}
depends_on = [helm_release.karpenter]
}
https://github.com/gavinbunney/terraform-provider-kubectl/issues/270#issuecomment-1625610825 works great, thanks a ton!
https://github.com/gavinbunney/terraform-provider-kubectl/issues/270#issuecomment-1625610825
This looks like a good alternative! I'll give that a shot thanks!
This seems to be a bug due to the way the provider handles discovery. I haven't investigated what exact behaviour in the api server changed in 1.27, but setting the GVK in a different way (see this commit) seems to fix it.
I've just opened https://github.com/alekc/terraform-provider-kubectl/pull/13 but on the @alekc's fork, since that's what I've been using for the past couple of months due to the low maintainer activity on this repo.
released in 2.0.2 (please raise an issue in case there are side effects, tested on kind from 1.19 to 1.27)
Nice work guys!
I was going to suggest to maybe write a quick tutorial here on how to compile your fork but I have checked it and see that you're already providing binary releases and the module is easily available under:
terraform {
required_version = ">= 0.13"
required_providers {
kubectl = {
source = "alekc/kubectl"
version = ">= 2.0.2"
}
}
On that note, I'm personally considering staying on the helm_release resource workaround.
It got me thinking after this incident that such a pretty popular provider, referenced by AWS terraform modules, can be abandoned and one day stop working completely.
If not now then in two years Kubernetes releases v1.38 or so, and everything will go south again.