kubernetes.client.exceptions.ApiException: (403) with sa default

[ric79]

Keywords

ApiException, serviceAccount

Problem

I wrote a very simple operator: https://gist.github.com/ric79/a650386d3cd8fade16b18827864efd7b#file-text-analyzer-py If I run locally, it is all ok

The deployment is without service account

$ more deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: textanalyzers-operator
  labels:
    app: textanalyzers-operator
spec:
  replicas: 1 # make sure to not have more than one replicas
  strategy:
    type: Recreate # make sure the old pod is being killed before the new pod is being created
  selector:
    matchLabels:
      app: textanalyzers-operator
  template:
    metadata:
      labels:
        app: textanalyzers-operator
    spec:
      containers:
      - name: textanalyzers-operator
        image: docker.io/mytest/text-analyzer
        imagePullPolicy: Never

When I create the deployment I get this POD error:

$ k get pods
NAME                                      READY   STATUS             RESTARTS      AGE
textanalyzers-operator-7758cb5647-sd9nn   0/1     CrashLoopBackOff   6 (18s ago)   6m22s

$ k get pod textanalyzers-operator-7758cb5647-sd9nn -o yaml | grep serviceAccount
  serviceAccount: default
  serviceAccountName: default

$ k logs textanalyzers-operator-7758cb5647-sd9nn
[2022-08-07 21:42:13,145] root                 [INFO    ] Trying CRD install
Traceback (most recent call last):
  File "/usr/local/bin/kopf", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kopf/cli.py", line 57, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kopf/cli.py", line 100, in run
    loaders.preload(
  File "/usr/local/lib/python3.9/site-packages/kopf/_cogs/helpers/loaders.py", line 41, in preload
    loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/root/app/text-analyzer.py", line 80, in <module>
    raise e
  File "/root/app/text-analyzer.py", line 75, in <module>
    api_instance.create_custom_resource_definition(text_analyzer_crd)
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/apiextensions_v1_api.py", line 66, in create_custom_resource_definition
    return self.create_custom_resource_definition_with_http_info(body, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/apiextensions_v1_api.py", line 157, in create_custom_resource_definition_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 275, in POST
    return self.request("POST", url,
  File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '4e670ba0-deb0-4468-8464-0a667ed402ab', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '28c64eea-9cd6-4023-97f0-86959af795ab', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'bbdf2318-ab9f-4816-a551-7333f90e3f4f', 'Date': 'Sun, 07 Aug 2022 21:42:13 GMT', 'Content-Length': '410'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"customresourcedefinitions.apiextensions.k8s.io is forbidden: User \"system:serviceaccount:mytest:default\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"group":"apiextensions.k8s.io","kind":"customresourcedefinitions"},"code":403}

I don't understand the reason. I tried also to use the sa textanalyzers-operator-account (see the previous link for the config), but the result is the same

Rr

[ric79]

I was able to run successfully the deployment using a cluster admin serviceaccount

$ k create -f - <<EOF
apiVersion: v1 
kind: ServiceAccount 
metadata: 
  name: my-operator 
EOF 

$ k create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRoleBinding 
metadata: 
  name: my-operator 
roleRef: 
  apiGroup: rbac.authorization.k8s.io 
  kind: ClusterRole 
  name: cluster-admin 
subjects: 
  - kind: ServiceAccount 
    name: my-operator 
    namespace: mytest 
EOF 

The pod error was: cannot create resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

So I modified the kopfexample-role-cluster (https://gist.github.com/ric79/a650386d3cd8fade16b18827864efd7b?permalink_comment_id=4259006#gistcomment-4259006) with

  # Framework: runtime observation of namespaces & CRDs (addition/deletion).
  - apiGroups: [apiextensions.k8s.io]
    resources: [customresourcedefinitions]
    # verbs: [list, watch]
    verbs: [get, list, watch, create, update, patch, delete]

and the error was solved