pulumi-kubernetes
pulumi-kubernetes copied to clipboard
pulumi
`kubernetes.helm.v4.Chart`: `failed to parse kubeconfig data` on preview
What happened?
Hi,
I would like to use the new kubernetes.helm.v4.Chart, to install an helm chart after the cluster creation.
But unfortunately if fails on the preview steps, as it expects the cluster to be already available:
error: Program failed with an unhandled exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1010, in do_rpc_call
return monitor.RegisterResource(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/grpc/_channel.py", line 1160, in __call__
return _end_unary_response_blocking(state, call, False, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-05-27T11:51:11.555748701+00:00", grpc_status:2, grpc_message:"configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn\'t get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" }"}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/pulumi-language-python-exec", line 191, in <module>
loop.run_until_complete(coro)
File "/usr/lib64/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 138, in run_in_stack
await run_pulumi_func(run)
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 52, in run_pulumi_func
await wait_for_rpcs()
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 114, in wait_for_rpcs
await task
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1015, in do_register
resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1012, in do_rpc_call
handle_grpc_error(exn)
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/settings.py", line 307, in handle_grpc_error
raise grpc_error_to_exception(exn)
Exception: configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
Instead the "older" v3 works:
kubernetes.helm.v3.Release(
"helm",
name="gitlab-runner",
namespace=namespace.metadata.name,
chart="gitlab-runner",
version="0.65.0",
repository_opts=kubernetes.helm.v3.RepositoryOptsArgs(
repo="https://charts.gitlab.io",
),
values={},
)
Example
kubernetes.helm.v4.Chart(
"helm",
name="gitlab-runner",
namespace=namespace.metadata.name,
chart="gitlab-runner",
version="0.65.0",
repository_opts=kubernetes.helm.v3.RepositoryOptsArgs(
repo="https://charts.gitlab.io",
),
values={},
)
Output of pulumi about
CLI
Version 3.117.0
Go Version go1.22.3
Go Compiler gc
Plugins
KIND NAME VERSION
resource command 0.11.1
resource hcloud 1.19.1
resource kubernetes 4.12.0
language python unknown
Host
OS fedora
Version 40
Arch x86_64
This project is written in python: executable='/usr/bin/python3' version='3.12.3'
Backend
Name f1e58f99868a
URL file:///data/gitlab-runner/.pulumi-local-states/dev
User root
Organizations
Token type personal
Dependencies:
NAME VERSION
ansible-core 2.16.6
argcomplete 3.3.0
kubernetes 29.0.0
pip 23.3.2
pulumi_command 0.11.1
pulumi_hcloud 1.19.1
pulumi_kubernetes 4.12.0
wheel 0.41.2
Pulumi locates its logs in /tmp by default
Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@yellowhat there's a problem with the provider's configuration. Are you setting the kubeconfig property, and if so could you elaborate on the specifics?
The Chart/v4 resource relies on a well-configured provider, and while we may make some improvement to the experience, I think we should focus on the provider's configuration in this case.
Sure.
my current setup involves creating the ~/.kube/config directory and fetching the kubeconfig from the master node before the actual kubernetes.Provider is created:
kubeconfig = local.Command(
f"{name}/kubeconfig",
create=Output.all(self.cp.ipv4_address).apply(lambda args: f"""
mkdir -p /root/.kube
scp {args[0]}:/etc/rancher/k3s/k3s.yaml /root/.kube/config
sed -e "s|https://.*|https://{args[0]}:6443|" \
-i /root/.kube/config
"""),
opts=ResourceOptions(
parent=child_opts.parent,
depends_on=[bootstrap],
),
)
self.kubeconfig = kubernetes.Provider(
name,
kubeconfig="/root/.kube/config",
opts=ResourceOptions(
parent=child_opts.parent,
depends_on=[kubeconfig],
),
)
Before and after pulumi preview the ~/.kube folder does not exist, it is created during the pulumi up process.
Thanks
Thanks for the explanation, it all makes sense. The provider is no doubt failing to find /root/.kube/config during preview, and enters a "degraded mode" that not all resource types understand well. While we explore some possible fixes, I would suggest that you either skip the previews, or create the kubeconfig by some other means such that it is available during preview.
@yellowhat another suggestion is to use the Run function, since it does run during preview. I assume that you'd want to download the kubeconfig during preview.
https://www.pulumi.com/registry/packages/command/api-docs/local/run/
Thanks for the explanation.
As you suggested I have run before pulumi preview:
mkdir -p ~/.kube
touch ~/.kube/config
and it is still trying to connect to he cluster (that does not exist yet):
pulumi:pulumi:Stack (hetzner-k8s-dev):
error: Program failed with an unhandled exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1010, in do_rpc_call
return monitor.RegisterResource(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/grpc/_channel.py", line 1160, in __call__
return _end_unary_response_blocking(state, call, False, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.12/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:
• set up the provider as per https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/
invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-05-29T06:06:50.986871168+00:00", grpc_status:2, grpc_message:"configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have: \n\n \t \xe2\x80\xa2 set up the provider as per https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/ \n\n invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/pulumi-language-python-exec", line 191, in <module>
loop.run_until_complete(coro)
File "/usr/lib64/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 138, in run_in_stack
await run_pulumi_func(run)
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 52, in run_pulumi_func
await wait_for_rpcs()
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/stack.py", line 114, in wait_for_rpcs
await task
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1015, in do_register
resp = await asyncio.get_event_loop().run_in_executor(None, do_rpc_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/resource.py", line 1012, in do_rpc_call
handle_grpc_error(exn)
File "/usr/local/lib/python3.12/site-packages/pulumi/runtime/settings.py", line 307, in handle_grpc_error
raise grpc_error_to_exception(exn)
Exception: configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:
• set up the provider as per https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/
invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
pulumi up --skip-preview seems to bypass the problem, but it is not a great alternative.
I did some investigation and, so far, haven't find an ideal solution. When the cluster is unreachable, the Chart resource cannot resolve kinds so as to produce their resource representation. Today the deployment fails, but if it were to skip rendering the children then they'd be marked for deletion.
At a technical level, we would need to move this check into each component provider's Construct function, and simply return the constructed component. We'd assumedly do that on Preview only.
I have similar issue. In my case provider is configured that way:
kubeconfig = Output.all(resource_group.name, cluster.name).apply(
lambda args: list_managed_cluster_admin_credentials(
resource_group_name=args[0],
resource_name=args[1],
).kubeconfigs[0].value
)
provider = Provider(
name,
kubeconfig=Output.secret(kubeconfig.apply(lambda enc: b64decode(enc).decode()))
)
I think it is common case when you want create cluster and install some base components f.e ArgoCD
I'm getting the exact same issue. Trying to create VPC, Cluster etc and then install ArgoCD.
const provider = new k8s.Provider("provider", {
kubeconfig: kubeConfigSecret
});
const argocd = new k8s.helm.v4.Chart("argo-cd", {
chart: "argo-cd",
version: "5.53.14",
namespace: "argo-cd",
repositoryOpts: {
repo: "https://argoproj.github.io/argo-helm",
},
valueYamlFiles: [
new pulumi.asset.FileAsset("./apps/argocd/values.yaml")
],
}, { dependsOn: argocdNamespace });
however I get the same error when trying to deploy...
Error: failed to register new resource argo-cd [kubernetes:helm.sh/v4:Chart]: 2 UNKNOWN: configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:
• set up the provider as per https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/
invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Could it just be that the provider falls over when using the : delimited list of kubeconfig files?
I'm having the same issue. I create a Kind cluster using a dynamic resource provider; the contents of the kubeconfig file become available after cluster creation. Because the cluster isn't created as part of preview, the Helm v4 provider fails during preview.
