workflows.argoproj.io is forbidden

[dommgifer]

kale: v0.7.0 kubeflow: v1.3

Run pipeline example from candies_sharing.ipynb

But got error:

Traceback (most recent call last):
  File "<string>", line 48, in <module>
  File "<string>", line 7, in sack
  File "/usr/local/lib/python3.6/dist-packages/kale/common/mlmdutils.py", line 533, in init_metadata
    mlmd_instance = MLMetadata()
  File "/usr/local/lib/python3.6/dist-packages/kale/common/mlmdutils.py", line 109, in __init__
    self.pod_namespace)
  File "/usr/local/lib/python3.6/dist-packages/kale/common/workflowutils.py", line 73, in get_workflow
    ARGO_WORKFLOWS_PLURAL, name)
  File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/apis/custom_objects_api.py", line 931, in get_namespaced_custom_object
    (data) = self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/apis/custom_objects_api.py", line 1031, in get_namespaced_custom_object_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 334, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 168, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 355, in request
    headers=headers)
  File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 231, in GET
    query_params=query_params)
  File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden

HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 31 May 2021 07:23:24 GMT', 'Content-Length': '441'})

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"workflows.argoproj.io \"candies-sharing-szcfg\" is forbidden: User \"system:serviceaccount:kubeflow-user-example-com:default-editor\" cannot get resource \"workflows\" in API group \"argoproj.io\" in the namespace \"kubeflow-user-example-com\"","reason":"Forbidden","details":{"name":"candies-sharing-szcfg","group":"argoproj.io","kind":"workflows"},"code":403}

Looks like permission is forbidden of Custom Resource workflows and API group argoproj.io .

Does anyone have any idea ?

[elikatsis]

Hi @dommgifer,

The steps need permissions to get their workflow. Try updating the permissions of the service account default-editor (that's the one your pods use) accordingly.

[fengredrum]

Hi @dommgifer,

The steps need permissions to get their workflow. Try updating the permissions of the service account default-editor (that's the one your pods use) accordingly.

@elikatsis Hi, I'm not familiar with K8s. Could you kindly specify how to update default-editor 's permissions?

[ConverJens]

@dommgifer @elikatsis @fengredrum I'm having the same issue using KFP 1.7: https://github.com/kubeflow-kale/kale/issues/367

[jacobmalmberg]

This can be solved by first creating a Role that allows access to the apigroup and then assigning the role to the default-editor serviceaccount:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow-role-kale-ns
rules:
- apiGroups: ["argoproj.io"]
  resources: ["pods", "workflows"]
  verbs: ["get", "watch", "patch"]

---

apiVersion: rbac.authorization.k8s.io/v1
# This role binding allows "jane" to read pods in the "default" namespace.
# You need to already have a Role named "pod-reader" in that namespace.
kind: RoleBinding
metadata:
  name: kale-ns
  namespace: YOUR-NS-HERE
subjects:
- kind: ServiceAccount
  name: default-editor

roleRef:
  # "roleRef" specifies the binding to a Role / ClusterRole
  kind: Role #this must be Role or ClusterRole
  name: workflow-role-kale-ns # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

---

[hieptran1812]

I faced the same problem. Have you fix this yet? @dommgifer