403 / KeyError: 'jinja_env' when accessing worker dashboard

[chrisroat]

What happened:

When accessing a scheduler dashboard on a GKE dask-gateway cluster, if I go to the "Info" panel and click on any "dashboard" link in the Services column, I get a 403. The traceback from the scheduler logs is:

tornado.application - ERROR - Uncaught exception in write_error
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/tornado/web.py", line 1681, in _execute
    result = self.prepare()
  File "/opt/conda/lib/python3.8/site-packages/jupyter_server/base/handlers.py", line 489, in prepare
    raise web.HTTPError(403)
tornado.web.HTTPError: HTTP 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/tornado/web.py", line 1217, in send_error
    self.write_error(status_code, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/jupyter_server/base/handlers.py", line 571, in write_error
    html = self.render_template('%s.html' % status_code, **ns)
  File "/opt/conda/lib/python3.8/site-packages/jupyter_server/base/handlers.py", line 502, in render_template
    template = self.get_template(name)
  File "/opt/conda/lib/python3.8/site-packages/jupyter_server/base/handlers.py", line 498, in get_template
    return self.settings['jinja2_env'].get_template(name)
KeyError: 'jinja2_env'

What you expected to happen:

I would learn what a worker dashboard looks like. For months I always get the 403, and just assumed it might go away someday during some upgrade.

Environment:

• Dask version: 2021.6.2
• Dask-Gateway: 0.9.0
• Python version: 3.8.10
• Operating System: Ubuntu 18.04
• Install method (conda, pip, source): conda (dask), pip (dask-gateway)

[chrisroat]

Example URL for the dashboard in case it helps:

https://XXX.YYY.ZZZ/services/dask-gateway/clusters/starmap.3a213abde5734e28a2eac9ea501b7f8b/proxy/8787/10.120.100.4/status

[hoang-innomize]

Is there anyone has any workaround/solution to fix this issue?