nebari-dev
[BUG] - Argo scheduled jobs fail with 403 permission error
Describe the bug
Server returned status code 403 with message: `workflows.argoproj.io is forbidden: User "system:serviceaccount:dev:argo-viewer" cannot create resource "workflows" in API group "argoproj.io" in the namespace "dev"`
Expected behavior
Runs smoothly
OS and architecture in which you are running Nebari
v2024.3.3
How to Reproduce the problem?
Run a notebook via scheduler
Command output
No response
Versions and dependencies used.
No response
Compute environment
None
Integrations
No response
Anything else?
No response
Our team is running into the same error as well (Nebari v2024.5.1).
@viniciusdc I tested two deployments with v2024.6.1rc3, one one this is happening on another it is not. The good news is it is happening on our internal dev deployment so it should be easy to troubleshoot.
It looks like users with analyst group only have the viewer privileges for workflows:
https://github.com/nebari-dev/nebari/blob/e997de84735b9b6eff8ea7323e979a76c6e56527/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/argo-workflows/main.tf#L89-L93
Indeed on the deployment on which everything works I was testing using an admin account, on deployment where it did not I had only a user account.
Summarising some offline discussions because I am not sure if everyone is aware of the multiple threads this was discussed in:
- @viniciusdc mentioned that this is the expected behaviour (that analysts cannot schedule notebooks)
- we agreed that in that case we need to improve the error message in
argo_jupyter_scheduler - separately @dharhas suggested to open an issue to think through a better permission role
The idea of the analyst and developer roles and what they each have access to is from an early clients usage of qhub and isn't very good in practice.
This is what I meant by we need to think through the permissions.
