spark-on-k8s-operator not working with disabled cluster role, clusterRoleBindings and webhook

[pm-nuance]

I am trying to install spark-operator but in a more restricted manner. Need my spark-operator to have access only in one namespace instead of entire cluster. Have disabled the installation of clusterRoles, clusterRoleBindings and Webhook. Spark-operator gets installed but when any job is submitted, no execturors or driver is created.

In the spark-operator logs, can see below error

E0120 16:06:45.429889 10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1beta2.ScheduledSparkApplication: failed to list *v1beta2.ScheduledSparkApplication: scheduledsparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:nuance-preprod-neap:spark-operator" cannot list resource "scheduledsparkapplications" in API group "sparkoperator.k8s.io" at the cluster scope: RBAC: clusterrole.rbac.authorization.k8s.io "spark-operator" not found E0120 16:07:14.871930 10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: sparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:nuance-preprod-neap:spark-operator" cannot list resource "sparkapplications" in API group "sparkoperator.k8s.io" at the cluster scope: RBAC: clusterrole.rbac.authorization.k8s.io "spark-operator" not found E0120 16:07:20.312479 10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:nuance-preprod-neap:spark-operator" cannot list resource "pods" in API group "" at the cluster scope: RBAC: clusterrole.rbac.authorization.k8s.io "spark-operator" not found E0120 16:07:41.413471 10 reflector.go:127] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1beta2.ScheduledSparkApplication: failed to list *v1beta2.ScheduledSparkApplication: scheduledsparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:nuance-preprod-neap:spark-operator" cannot list resource "scheduledsparkapplications" in API group "sparkoperator.k8s.io" at the cluster scope: RBAC: clusterrole.rbac.authorization.k8s.io "spark-operator" not found

Is there any way to install spark-operator but with webhook disabled and no clusterRoles??

[mgyboom]

Same situation. @liyinan926

--set rbac.createClusterRole=false

spark-operator.log:

E0711 10:35:16.140557      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.ScheduledSparkApplication: failed to list *v1beta2.ScheduledSparkApplication: scheduledsparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "scheduledsparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"
E0711 10:35:17.015872      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: sparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "sparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"
E0711 10:35:52.124576      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.ScheduledSparkApplication: failed to list *v1beta2.ScheduledSparkApplication: scheduledsparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "scheduledsparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"
E0711 10:35:57.936653      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: sparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "sparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"
E0711 10:35:59.777570      10 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "pods" in API group "" in the namespace "ibond-scheduler"
E0711 10:36:22.312129      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.ScheduledSparkApplication: failed to list *v1beta2.ScheduledSparkApplication: scheduledsparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "scheduledsparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"

[yangjf2019]

Hi, has this problem been solved yet? I'm experiencing the same problem. @mgyboom @liyinan926

Same situation. @liyinan926

--set rbac.createClusterRole=false

spark-operator.log:

E0711 10:35:16.140557      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.ScheduledSparkApplication: failed to list *v1beta2.ScheduledSparkApplication: scheduledsparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "scheduledsparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"
E0711 10:35:17.015872      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: sparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "sparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"
E0711 10:35:52.124576      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.ScheduledSparkApplication: failed to list *v1beta2.ScheduledSparkApplication: scheduledsparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "scheduledsparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"
E0711 10:35:57.936653      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: sparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "sparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"
E0711 10:35:59.777570      10 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "pods" in API group "" in the namespace "ibond-scheduler"
E0711 10:36:22.312129      10 reflector.go:127] pkg/client/informers/externalversions/factory.go:119: Failed to watch *v1beta2.ScheduledSparkApplication: failed to list *v1beta2.ScheduledSparkApplication: scheduledsparkapplications.sparkoperator.k8s.io is forbidden: User "system:serviceaccount:ibond-scheduler:spark-operator" cannot list resource "scheduledsparkapplications" in API group "sparkoperator.k8s.io" in the namespace "ibond-scheduler"

[lfreinag]

Hello there!

I have the same problem as @mgyboom and @liyinan926. 🙏 Any solution to this?

[lfreinag]

Not sure if this helps but it turned out to be that the service account and the referenced name were different. The values file actually manages to define the name of the service account. So I did that and it all started to work.

Hope this helps the others here. 😉

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.