argo-workflows icon indicating copy to clipboard operation
argo-workflows copied to clipboard

Published 20 hours ago • verified publisher icon argoproj

RBAC enabled but not restricting user

Open LoricAndre opened this issue 2 years ago • 11 comments

Checklist

  • [x] Double-checked my configuration.
  • [x] Tested using the latest version.
  • [x] Used the Emissary executor.

Summary

What happened/what you expected to happen? After setting up SSO and RBAC, SSO is working and assigning me the right ServiceAccount, but the rights I have on the server are more open than the associated role gives. This is my RBAC SA/Role/RoleBinding:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-workflow-default-user-login
  namespace: {{ .Release.Namespace }}
  annotations:
    workflows.argoproj.io/rbac-rule: "'my_group' in groups"
    workflows.argoproj.io/rbac-rule-precedence: "0"  # if the user is not in my_group, he should not and cannot access the server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-workflow-reader
  namespace: {{ .Release.Namespace }}
rules:
  - apiGroups:
      - argoproj.io
    resources:
      - workflows
      - workfloweventbindings
      - workflowtemplates
      - cronworkflows
      - cronworkflows/finalizers
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-workflow-reader
  namespace: {{ .Release.Namespace }}
subjects:
- kind: ServiceAccount
  name: argo-workflow-default-user-login
roleRef:
  kind: Role
  name: argo-workflow-reader
  apiGroup: rbac.authorization.k8s.io

What version are you running? v3.3.1

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

Any workflow

# Logs for the workflow controller:

time="2022-04-04T12:05:38.101Z" level=info msg="Update leases 200"
time="2022-04-04T12:05:41.374Z" level=info msg="Watch workflowtemplates 200"
time="2022-04-04T12:05:43.117Z" level=info msg="Get leases 200"
time="2022-04-04T12:05:43.127Z" level=info msg="Update leases 200"
time="2022-04-04T12:05:48.138Z" level=info msg="Get leases 200"
time="2022-04-04T12:05:48.147Z" level=info msg="Update leases 200"
time="2022-04-04T12:05:52.939Z" level=info msg="List workflows 200"
time="2022-04-04T12:05:52.939Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2022-04-04T12:05:53.160Z" level=info msg="Get leases 200"
time="2022-04-04T12:05:53.169Z" level=info msg="Update leases 200"
time="2022-04-04T12:05:55.580Z" level=info msg="List workflowtasksets 404"
E0404 12:05:55.580889       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.WorkflowTaskSet: failed to list *v1alpha1.WorkflowTaskSet: the server could not find the requested resource (get workflowtasksets.argoproj.io)
time="2022-04-04T12:05:57.967Z" level=info msg="Watch configmaps 200"
time="2022-04-04T12:05:58.179Z" level=info msg="Get leases 200"
time="2022-04-04T12:05:58.189Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:03.205Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:03.216Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:08.225Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:08.237Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:13.249Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:13.257Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:18.272Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:18.284Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:23.294Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:23.304Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:28.316Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:28.328Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:32.497Z" level=info msg="Watch clusterworkflowtemplates 200"
time="2022-04-04T12:06:33.341Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:33.355Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:33.671Z" level=info msg="Watch workflows 200"
time="2022-04-04T12:06:38.367Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:38.378Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:43.391Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:43.405Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:47.182Z" level=info msg="List workflowtasksets 404"
E0404 12:06:47.182314       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.WorkflowTaskSet: failed to list *v1alpha1.WorkflowTaskSet: the server could not find the requested resource (get workflowtasksets.argoproj.io)
time="2022-04-04T12:06:48.415Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:48.427Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:52.939Z" level=info msg="List workflows 200"
time="2022-04-04T12:06:52.939Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2022-04-04T12:06:53.440Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:53.451Z" level=info msg="Update leases 200"
time="2022-04-04T12:06:58.404Z" level=info msg="Watch configmaps 200"
time="2022-04-04T12:06:58.463Z" level=info msg="Get leases 200"
time="2022-04-04T12:06:58.473Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:03.485Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:03.497Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:08.512Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:08.522Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:13.535Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:13.558Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:18.568Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:18.579Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:19.450Z" level=info msg="Watch pods 200"
time="2022-04-04T12:07:19.584Z" level=info msg="List workflowtasksets 404"
E0404 12:07:19.584489       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.WorkflowTaskSet: failed to list *v1alpha1.WorkflowTaskSet: the server could not find the requested resource (get workflowtasksets.argoproj.io)
time="2022-04-04T12:07:23.592Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:23.605Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:28.613Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:28.623Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:33.635Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:33.646Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:38.657Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:38.668Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:43.680Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:43.690Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:48.702Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:48.725Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:52.935Z" level=info msg="List workflows 200"
time="2022-04-04T12:07:52.935Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2022-04-04T12:07:53.735Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:53.758Z" level=info msg="Update leases 200"
time="2022-04-04T12:07:57.285Z" level=info msg="Alloc=6497 TotalAlloc=7311249 Sys=74065 NumGC=3513 Goroutines=202"
time="2022-04-04T12:07:58.772Z" level=info msg="Get leases 200"
time="2022-04-04T12:07:58.782Z" level=info msg="Update leases 200"
time="2022-04-04T12:08:03.342Z" level=info msg="List workflowtasksets 404"
E0404 12:08:03.342876       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.WorkflowTaskSet: failed to list *v1alpha1.WorkflowTaskSet: the server could not find the requested resource (get workflowtasksets.argoproj.io)
time="2022-04-04T12:08:03.794Z" level=info msg="Get leases 200"
time="2022-04-04T12:08:03.813Z" level=info msg="Update leases 200"
time="2022-04-04T12:08:08.827Z" level=info msg="Get leases 200"
time="2022-04-04T12:08:08.837Z" level=info msg="Update leases 200"
time="2022-04-04T12:08:13.848Z" level=info msg="Get leases 200"
time="2022-04-04T12:08:13.857Z" level=info msg="Update leases 200"
time="2022-04-04T12:08:14.561Z" level=info msg="Queueing Succeeded workflow argo-workflow/lovely-python-wjg8h for delete in 1m38s"
time="2022-04-04T12:10:45.411Z" level=info msg="List workflowtasksets 404"
E0404 12:10:45.411898       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.WorkflowTaskSet: failed to list *v1alpha1.WorkflowTaskSet: the server could not find the requested resource (get workflowtasksets.argoproj.io)
time="2022-04-04T12:10:49.588Z" level=info msg="Get leases 200"
time="2022-04-04T12:10:49.601Z" level=info msg="Update leases 200"
time="2022-04-04T12:10:52.935Z" level=info msg="List workflows 200"
time="2022-04-04T12:10:52.935Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2022-04-04T12:10:54.612Z" level=info msg="Get leases 200"
time="2022-04-04T12:10:54.624Z" level=info msg="Update leases 200"
time="2022-04-04T12:10:59.645Z" level=info msg="Get leases 200"
time="2022-04-04T12:10:59.657Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:04.671Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:04.682Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:09.694Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:09.722Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:14.741Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:14.751Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:19.764Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:19.781Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:24.794Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:24.804Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:29.818Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:29.831Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:34.839Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:34.850Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:39.863Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:39.872Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:40.162Z" level=info msg="List workflowtasksets 404"
E0404 12:11:40.163143       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.WorkflowTaskSet: failed to list *v1alpha1.WorkflowTaskSet: the server could not find the requested resource (get workflowtasksets.argoproj.io)
time="2022-04-04T12:11:44.888Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:44.900Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:49.910Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:49.921Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:52.936Z" level=info msg="List workflows 200"
time="2022-04-04T12:11:52.936Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2022-04-04T12:11:54.936Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:54.947Z" level=info msg="Update leases 200"
time="2022-04-04T12:11:59.958Z" level=info msg="Get leases 200"
time="2022-04-04T12:11:59.969Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:04.983Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:04.997Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:10.009Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:10.019Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:15.035Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:15.045Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:20.055Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:20.067Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:25.087Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:25.098Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:27.887Z" level=info msg="List workflowtasksets 404"
E0404 12:12:27.887182       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.WorkflowTaskSet: failed to list *v1alpha1.WorkflowTaskSet: the server could not find the requested resource (get workflowtasksets.argoproj.io)
time="2022-04-04T12:12:30.110Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:30.121Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:35.135Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:35.157Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:40.168Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:40.177Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:45.188Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:45.198Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:47.504Z" level=info msg="Watch clusterworkflowtemplates 200"
time="2022-04-04T12:12:50.215Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:50.226Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:52.935Z" level=info msg="List workflows 200"
time="2022-04-04T12:12:52.935Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2022-04-04T12:12:55.237Z" level=info msg="Get leases 200"
time="2022-04-04T12:12:55.246Z" level=info msg="Update leases 200"
time="2022-04-04T12:12:57.284Z" level=info msg="Alloc=7360 TotalAlloc=7316549 Sys=74065 NumGC=3515 Goroutines=202"
time="2022-04-04T12:13:00.259Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:00.272Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:05.282Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:05.297Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:10.308Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:10.319Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:11.675Z" level=info msg="Watch workflows 200"
time="2022-04-04T12:13:15.330Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:15.339Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:20.352Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:20.362Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:22.363Z" level=info msg="List workflowtasksets 404"
E0404 12:13:22.363688       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.WorkflowTaskSet: failed to list *v1alpha1.WorkflowTaskSet: the server could not find the requested resource (get workflowtasksets.argoproj.io)
time="2022-04-04T12:13:25.376Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:25.388Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:30.397Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:30.406Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:35.420Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:35.430Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:40.440Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:40.451Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:45.465Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:45.475Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:48.542Z" level=info msg="Watch workflows 200"
time="2022-04-04T12:13:50.485Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:50.504Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:52.938Z" level=info msg="List workflows 200"
time="2022-04-04T12:13:52.938Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=
time="2022-04-04T12:13:55.523Z" level=info msg="Get leases 200"
time="2022-04-04T12:13:55.534Z" level=info msg="Update leases 200"
time="2022-04-04T12:13:58.420Z" level=info msg="Watch cronworkflows 200"
time="2022-04-04T12:14:00.544Z" level=info msg="Get leases 200"
time="2022-04-04T12:14:00.555Z" level=info msg="Update leases 200"
time="2022-04-04T12:14:02.377Z" level=info msg="Watch workflowtemplates 200"

# Logs from the workflow server:

time="2022-04-04T12:04:04.762Z" level=info msg="not enabling pprof debug endpoints"
time="2022-04-04T12:04:04.764Z" level=info authModes="[sso]" baseHRef=/ managedNamespace= namespace=argo-workflow secure=false
time="2022-04-04T12:04:04.764Z" level=warning msg="You are running in insecure mode. Learn how to enable transport layer security: https://argoproj.github.io/argo-workflows/tls/"
time="2022-04-04T12:04:04.764Z" level=info msg="config map" name=argo-workflow-gtw-argo-workflows-workflow-controller-configmap
time="2022-04-04T12:04:05.435Z" level=info msg="SSO configuration" clientId="{{argo-workflow-secret-infra-argo-workflow-oidc} client_id <nil>}" insecureSkipVerify=false issuer="****************" issuerAlias=DISABLED redirectUrl="***********************" scopes="[groups openid profile email openid]"
time="2022-04-04T12:04:05.537Z" level=info msg="SSO enabled"
time="2022-04-04T12:04:05.574Z" level=info msg="Starting Argo Server" instanceID= version=v3.3.1
time="2022-04-04T12:04:05.574Z" level=info msg="Creating DB session"
time="2022-04-04T12:04:05.792Z" level=info msg="Node status offloading config" ttl=5m0s
time="2022-04-04T12:04:05.792Z" level=info msg="Creating event controller" asyncDispatch=false operationQueueSize=16 workerCount=4
time="2022-04-04T12:04:05.808Z" level=info msg="GRPC Server Max Message Size, MaxGRPCMessageSize, is set" GRPC_MESSAGE_SIZE=104857600
time="2022-04-04T12:04:05.809Z" level=info msg="Argo Server started successfully on http://localhost:2746"
time="2022-04-04T12:04:32.976Z" level=info msg="selected SSO RBAC service account for user" email=***************** loginServiceAccount=argo-workflow-default-user-login serviceAccount=argo-workflow-default-user-login ssoDelegated=false ssoDelegationAllowed=false subject=c3235165-381d-4ddf-9a3f-e5d20f8dc4ac
time="2022-04-04T12:04:32.996Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2022-04-04T12:04:32Z" grpc.time_ms=24.762 span.kind=server system=grpc
time="2022-04-04T12:04:34.375Z" level=info msg="selected SSO RBAC service account for user" email=********************* loginServiceAccount=argo-workflow-default-user-login serviceAccount=argo-workflow-default-user-login ssoDelegated=false ssoDelegationAllowed=false subject=c3235165-381d-4ddf-9a3f-e5d20f8dc4ac
time="2022-04-04T12:04:34.387Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetWorkflowTemplate grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2022-04-04T12:04:34Z" grpc.time_ms=16.029 span.kind=server system=grpc
time="2022-04-04T12:04:39.266Z" level=info msg="selected SSO RBAC service account for user" email=******************* loginServiceAccount=argo-workflow-default-user-login serviceAccount=argo-workflow-default-user-login ssoDelegated=false ssoDelegationAllowed=false subject=c3235165-381d-4ddf-9a3f-e5d20f8dc4ac
time="2022-04-04T12:04:39.280Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetWorkflowTemplate grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2022-04-04T12:04:39Z" grpc.time_ms=18.811 span.kind=server system=grpc
time="2022-04-04T12:04:39.731Z" level=info msg="selected SSO RBAC service account for user" email=********************* loginServiceAccount=argo-workflow-default-user-login serviceAccount=argo-workflow-default-user-login ssoDelegated=false ssoDelegationAllowed=false subject=c3235165-381d-4ddf-9a3f-e5d20f8dc4ac
time="2022-04-04T12:04:39.732Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2022-04-04T12:04:39Z" grpc.time_ms=6.189 span.kind=server system=grpc
time="2022-04-04T12:04:45.469Z" level=info msg="selected SSO RBAC service account for user" email=******************** loginServiceAccount=argo-workflow-default-user-login serviceAccount=argo-workflow-default-user-login ssoDelegated=false ssoDelegationAllowed=false subject=c3235165-381d-4ddf-9a3f-e5d20f8dc4ac

# If the workflow's pods have not been created, you can skip the rest of the diagnostics.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

LoricAndre avatar Apr 04 '22 12:04 LoricAndre

Can you be more specific? For example, do you go into the UI, and are allowed to update a workflow?

What is your Kubernetes provider? Does it both support and have RBAC enabled correctly? E.g. Docker for Desktop does not support RBAC. Certain cloud configurations don't either.

alexec avatar Apr 04 '22 14:04 alexec

Hi, thank you for that quick answer.

I can get into the UI without issues and the SSO SA is correctly assigned, which I can see in the User tab. The issue is that once logged in, I can create and submit workflows without any error.

My Kubernetes provider is Azure AKS, and RBAC is enabled and used successfully in other projects.

LoricAndre avatar Apr 05 '22 07:04 LoricAndre

I think this is most likely to be mis-configuration, so I'm don't want to invest too much time until we've checked that.

  • Can you confirm that the correct service account in being recieved by the Kubernetes API Server by checking your logs.
  • Can you double-check the service account using kubectl auth can-i?

If that fails, please book 30m via the new issue link.

alexec avatar Apr 05 '22 17:04 alexec

Attempted to repro, failed:

argo-server | time="2022-04-07T14:38:12.194Z" level=info msg="selected SSO RBAC service account for user" [email protected] loginServiceAccount=nothing serviceAccount=nothing ssoDelegated=false ssoDelegationAllowed=false subject=Cg0wLTM4NS0yODA4OS0wEgRtb2Nr
argo-server | time="2022-04-07T14:38:12.206Z" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflows.argoproj.io is forbidden: User \"system:serviceaccount:argo:nothing\" cannot list resource \"workflows\" in API group \"argoproj.io\" in the namespace \"argo\"" grpc.code=PermissionDenied grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2022-04-07T14:38:12-07:00" grpc.time_ms=13.813 span.kind=server system=grpc

alexec avatar Apr 07 '22 21:04 alexec 

 kubectl auth can-i create workflows --as=system:serviceaccount:argo:nothing -n argo
no

alexec avatar Apr 07 '22 21:04 alexec

I think this is most likely to be mis-configuration, so I'm don't want to invest too much time until we've checked that.

  • Can you confirm that the correct service account in being recieved by the Kubernetes API Server by checking your logs.

It is, my email is associated with: loginServiceAccount=argo-workflow-default-user-login serviceAccount=argo-workflow-default-user-login ssoDelegated=false ssoDelegationAllowed=false

  • Can you double-check the service account using kubectl auth can-i?

Impersonation is disabled on the cluster, I cannot test this.

LoricAndre avatar Apr 12 '22 14:04 LoricAndre 

time="2023-07-13T09:08:42.134Z" level=info msg="selected SSO RBAC service account for user" email=****@****.** loginServiceAccount=tmp-sso-argo-workflows serviceAccount=tmp-sso-argo-workflows ssoDelegated=false ssoDelegationAllowed=false subject=**********
time="2023-07-13T09:08:42.135Z" level=info msg="selected SSO RBAC service account for user" email=****@****.** loginServiceAccount=tmp-sso-argo-workflows serviceAccount=tmp-sso-argo-workflows ssoDelegated=false ssoDelegationAllowed=false subject=**********
time="2023-07-13T09:08:42.139Z" level=info msg="tracking UI usage️️" email=****@****.**  name=openedSensorList subject=**********
time="2023-07-13T09:08:42.139Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=CollectEvent grpc.service=info.InfoService grpc.start_time="2023-07-13T09:08:42Z" grpc.time_ms=6.564 span.kind=server system=grpc

kubectl auth can-i list sensors --as=system:serviceaccount:argo:tmp-sso-argo-workflows -n argo
no

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo:operator
rules:
  - apiGroups:
      - argoproj.io
    resources:
      - workflowtemplates
    resourceNames:
      - ci-k8s
      - ci-protobuf
      - ci-python
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - pods/log
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: "tmp-sso-argo:operator"
subjects:
- kind: ServiceAccount
  name: tmp-sso-argo-workflows
  namespace: argo
roleRef:
  kind: Role
  name: argo:operator
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: "tmp-sso-argo-workflows"
  annotations:
    workflows.argoproj.io/rbac-rule: "'*****:****' in groups"
    workflows.argoproj.io/rbac-rule-precedence: "0"
secrets:
    - name: github-sso-argo-workflows

qtheya avatar Jul 13 '23 09:07 qtheya

@qtheya Does your comment(https://github.com/argoproj/argo-workflows/issues/8310#issuecomment-1633869767) mean that you succeeded to reproduce the bug?

umi0410 avatar Jul 20 '23 14:07 umi0410

@qtheya Does your comment(https://github.com/argoproj/argo-workflows/issues/8310#issuecomment-1633869767) mean that you succeeded to reproduce the bug?

Yes

qtheya avatar Jul 20 '23 14:07 qtheya