Celery k8s run launcher tries to launch job in "default" k8s namespace

[maxfirman]

Dagster version

0.5.15

What's the issue?

I've installed Dagster using the helm chart and enabled the celery run launcher and rabbitmq.

When I execute any task from Dagit I get the following error because it tries to launch a job in the "default" namespace and the service account does not have the appropriate permissions:

Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '9f07d8a2-fcb2-4140-95a7-3c0ea51b3f8a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'ca849dcf-7fc7-4476-a909-f559b5cd4902', 'X-Kubernetes-Pf-Prioritylevel-Uid': '9ce2dd78-1664-4343-ba13-26e90250e7de', 'Date': 'Fri, 15 Jul 2022 09:30:16 GMT', 'Content-Length': '314'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:dp-dagster:dagster\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}


  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/utils.py", line 49, in _fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 21, in launch_pipeline_execution
    return _launch_pipeline_execution(graphene_info, execution_params)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 53, in _launch_pipeline_execution
    run = do_launch(graphene_info, execution_params, is_reexecuted)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/execution/launch_execution.py", line 41, in do_launch
    workspace=graphene_info.context,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 1681, in submit_run
    SubmitRunContext(run, workspace=workspace)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/run_coordinator/default_run_coordinator.py", line 32, in submit_run
    self._instance.launch_run(pipeline_run.run_id, context.workspace)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 1733, in launch_run
    self._run_launcher.launch_run(LaunchRunContext(pipeline_run=run, workspace=workspace))
  File "/usr/local/lib/python3.7/site-packages/dagster_celery_k8s/launcher.py", line 240, in launch_run
    self._batch_api.create_namespaced_job(body=job, namespace=job_namespace)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_job
    return self.create_namespaced_job_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/batch_v1_api.py", line 323, in create_namespaced_job_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
    _preload_content, _request_timeout, _host)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 397, in request
    body=body)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 281, in POST
    body=body)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)

What did you expect to happen?

I would expect it to launch jobs in the helm chart's release k8s namespace rather than the "default" k8s namespace

How to reproduce?

Install helm chart and enable celery k8s run launcher and rabbitmq and try to run any job.

Deployment type

Dagster Helm chart

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[Eugst]

+1

[gmontanola]

https://github.com/dagster-io/dagster/pull/9235

AFAIK this should do.

[Milias]

I believe this issue can be closed, fixed in #12911.