Cannot preview RBAC resources when user is not a cluster admin

[ChristianRaoulis]

What happened?

I'm trying to deploy the RabbitMQ Helm Chart of Bitnami using the helm.v3.Chart class but for some reason pulumi tried to create a RoleBinding before the is Role created.

Relevant log from pulumi preview:

Previewing update (stack):
 +  kubernetes:helm.sh/v3:Chart rabbitmq create 
 +  kubernetes:core/v1:Service core-services/rabbitmq-headless create 
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create 
 +  kubernetes:core/v1:ServiceAccount core-services/rabbitmq create 
 +  kubernetes:core/v1:Service core-services/rabbitmq create 
 +  kubernetes:core/v1:Service core-services/rabbitmq-headless create 
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create Retry #0; creation failed: roles.rbac.authorization.k8s.io "rabbitmq-endpoint-reader" not found
 +  kubernetes:rbac.authorization.k8s.io/v1:Role core-services/rabbitmq-endpoint-reader create 
 +  kubernetes:core/v1:ServiceAccount core-services/rabbitmq create 
 +  kubernetes:core/v1:Secret core-services/rabbitmq-config create 
 +  kubernetes:core/v1:Service core-services/rabbitmq create 
 +  kubernetes:networking.k8s.io/v1:Ingress core-services/rabbitmq create 
 +  kubernetes:core/v1:Secret core-services/rabbitmq-config create 
 +  kubernetes:apps/v1:StatefulSet core-services/rabbitmq create 
 +  kubernetes:rbac.authorization.k8s.io/v1:Role core-services/rabbitmq-endpoint-reader create 
 +  kubernetes:apps/v1:StatefulSet core-services/rabbitmq create 
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create Retry #1; creation failed: roles.rbac.authorization.k8s.io "rabbitmq-endpoint-reader" not found
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create Retry #2; creation failed: roles.rbac.authorization.k8s.io "rabbitmq-endpoint-reader" not found
 +  kubernetes:networking.k8s.io/v1:Ingress core-services/rabbitmq create 
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create Retry #3; creation failed: roles.rbac.authorization.k8s.io "rabbitmq-endpoint-reader" not found
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create Retry #4; creation failed: roles.rbac.authorization.k8s.io "rabbitmq-endpoint-reader" not found
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create Retry #5; creation failed: roles.rbac.authorization.k8s.io "rabbitmq-endpoint-reader" not found
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create error: Preview failed: resource "urn:pulumi:testing::softwarefactory::kubernetes:core/v1:Namespace$softwarefactory:rabbitmq$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:RoleBinding::core-services/rabbitmq-endpoint-reader" was not successfully created by the Kubernetes API server : roles.rbac.authorization.k8s.io "rabbitmq-endpoint-reader" not found
 +  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding core-services/rabbitmq-endpoint-reader create 1 error
Diagnostics:
  kubernetes:rbac.authorization.k8s.io/v1:RoleBinding (core-services/rabbitmq-endpoint-reader):
    error: Preview failed: resource "urn:pulumi:testing::softwarefactory::kubernetes:core/v1:Namespace$softwarefactory:rabbitmq$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:RoleBinding::core-services/rabbitmq-endpoint-reader" was not successfully created by the Kubernetes API server : roles.rbac.authorization.k8s.io "rabbitmq-endpoint-reader" not found

Example

import * as kubernetes from "@pulumi/kubernetes";

new kubernetes.helm.v3.Chart("rabbitmq", {
         chart:     "rabbitmq",
         fetchOpts: {
            repo: "https://charts.bitnami.com/bitnami",
         },
         values: {
            auth: {
               username: "admin",
               password: "admin"
            },
         },
});

Output of pulumi about

CLI          
Version      3.107.0
Go Version   go1.22.0
Go Compiler  gc

Plugins
NAME        VERSION
command     0.10.0
kubernetes  4.10.0
nodejs      unknown
postgresql  3.11.0
random      4.16.1

Host
OS       Microsoft Windows 11 Enterprise
Version  10.0.22621 Build 22621
Arch     x86_64

This project is written in nodejs: executable='C:\Users\A92615470\AppData\Local\pnpm\node.exe' version='v20.11.0'

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[blampe]

Hi @ChristianRaoulis, this is a known issue with the v3 Chart resource, specifically https://github.com/pulumi/pulumi-kubernetes/issues/2227.

We're currently working on a v4 Chart which will address this and many other issues https://github.com/pulumi/pulumi-kubernetes/issues/2847 -- please stay tuned!

[EronWright]

I would suggest using Release resource as a workaround for now. It is effective because previews of a Release doesn't perform a dry-run.

Even if roles were created before bindings, the role wouldn't actually exist during preview and so the preview logic would need to be special-cased. Typically, Kubernetes accepts RoleBinding objects that refer to a non-existent Role. But I think I found an explanation here, it is that @ChristianRaoulis perhaps doesn't have admin access to the cluster. Could you confirm?

Update: I am able to repro the issue when I use a non-admin account. Here's how:

# admin.yaml - grant 'admin' role to 'myuser' in 'default' namespace, to be able to create bindings.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myuser-default-admin
  namespace: default
subjects:
- kind: User
  name: myuser
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io

# rabbitmq.yaml - grant 'foobar' role (non-existent) to 'rabbitmq' in 'default' namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rabbitmq-foobar
  namespace: default
subjects:
- kind: ServiceAccount
  name: rabbitmq
  namespace: default
roleRef:
  kind: ClusterRole
  name: foobar
  apiGroup: rbac.authorization.k8s.io

❯ kubectl apply -f admin.yaml
rolebinding.rbac.authorization.k8s.io/myuser-default-admin created

❯ kubectl apply -f rabbitmq.yaml --as myuser
Error from server (NotFound): error when creating "rabbitmq.yaml": clusterroles.rbac.authorization.k8s.io "foobar" not found

❯ kubectl apply -f rabbitmq.yaml
rolebinding.rbac.authorization.k8s.io/rabbitmq-foobar created

[EronWright]

I see two possible improvements here:

1. In preview, use client-side validation only for bindings, no server-side dry-run.
2. Enhance the ordering logic to create roles before bindings, to avoid retries that would otherwise occur due to the race.

[EronWright]

@ChristianRaoulis I notice that the resource URN has an unusual value:

urn:pulumi:testing::softwarefactory::kubernetes:core/v1:Namespace$softwarefactory:rabbitmq$kubernetes:helm.sh/v3:Chart$kubernetes:rbac.authorization.k8s.io/v1:RoleBinding::core-services/rabbitmq-endpoint-reader

It seems like a core-services namespace resource is involved, and maybe being used as a parent? Please use dependsOn rather than parent. It is a separate problem but I felt I should mention it.

[ChristianRaoulis]

Hi @ChristianRaoulis, this is a known issue with the v3 Chart resource, specifically #2227.

We're currently working on a v4 Chart which will address this and many other issues #2847 -- please stay tuned!

Is there a time horizon when Chart V4 will be usable? Days, weeks, months?

[ChristianRaoulis]

I would suggest using Release resource as a workaround for now. It is effective because previews of a Release doesn't perform a dry-run.

Thank you :)

Even if roles were created before bindings, the role wouldn't actually exist during preview and so the preview logic would need to be special-cased. Typically, Kubernetes accepts RoleBinding objects that refer to a non-existent Role. But I think I found an explanation here, it is that @ChristianRaoulis perhaps doesn't have admin access to the cluster. Could you confirm?

Confirmed 👍

It seems like a core-services namespace resource is involved, and maybe being used as a parent? Please use dependsOn rather than parent.

I passed the namespace to the chart using the namespace: namespace.metadata.name prop but the result is the same :)

[blampe]

@ChristianRaoulis the new Chart v4 resource is ready for beta testing if you'd like to try it out! Feel free to reach out via email (in my GH profile) or in the Pulumi community Slack if that's something you're interested in.

[EronWright]

I do not think that Chart v4 will solve this issue, because the problem isn't related to ordering, it is due to the special case behavior of RoleBinding when you're a non-admin. The only real fix would be to add some special-case logic to the provider, e.g. to use client-side validation only for RoleBinding during preview, or to swallow the error.

[ChristianRaoulis]

I do not think that Chart v4 will solve this issue, because the problem isn't related to ordering, it is due to the special case behavior of RoleBinding when you're a non-admin. The only real fix would be to add some special-case logic to the provider, e.g. to use client-side validation only for RoleBinding during preview, or to swallow the error.

I can confirm that.

I ran into the role not found error again but this time with a role that i create myself in the same preview.

const role = new Role('role', {
   metadata: {
      name:      'service-account-role',
      namespace: namespace.metadata.name,
   },
   rules:    [
      {
         apiGroups: [""],
         resources: [
            "secrets",
         ],
         verbs:     [
            "get",
            "update",
         ],
      },
   ],
});
const serviceAccount = new ServiceAccount("service-account", {
   metadata: {
      name:      "service1",
      namespace: namespace.metadata.name,
   },
});
const roleBinding = new RoleBinding('role-binding', {
   metadata: {
      name:      'role-binding',
      namespace: namespace.metadata.name,
   },
   roleRef:  {
      apiGroup: "rbac.authorization.k8s.io",
      kind:     "Role",
      name:     role.metadata.name,
   },
   subjects: [
      {
         kind: "ServiceAccount",
         name: serviceAccount.metadata.name,
      },
   ],
}, {parent: serviceAccount});

Is there any workaround i could use for now?