RuntimeError when generating cloud init config for Packer (GCPCluster)

[alexwalterbos]

Describe the issue:

I'm trying to build a custom OS image for my GCP cluster with packer. I'm using these steps: https://cloudprovider.dask.org/en/latest/packer.html

The util get_cloud_init() fails, seemingly on some underlying async logic.

Minimal Complete Verifiable Example:

from dask_cloudprovider.gcp import GCPCluster

if __name__ == '__main__':
    cic = GCPCluster.get_cloud_init(
        projectid='valid-project-id-that-works-for-cluster-creation',
        zone='europe-xxxx-x',
        machine_type='c2-standard-8', # but occurs with other machine types as well
        env_vars={
            "EXTRA_PIP_PACKAGES": "joblib==1.3.2 sqlalchemy==2.0.13 scikit-learn==1.3.2 pandas==2.1.4 numpy==1.26.2 nltk==3.6.7"},
    )
    print(cic)

Anything else we need to know?: Error traceback:

/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/bin/python3 /home/alexwalterbos/repos/magnet.me/services/recommender/similarity/learn/esco/classifiers/get_cloud_config.py
Traceback (most recent call last):
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/learn/esco/classifiers/get_cloud_config.py", line 6, in <module>
    cic = GCPCluster.get_cloud_init(
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/dask_cloudprovider/generic/vmcluster.py", line 365, in get_cloud_init
    cluster = cls(*args, asynchronous=True, **kwargs)
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/dask_cloudprovider/gcp/instances.py", line 634, in __init__
    super().__init__(debug=debug, **kwargs)
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/dask_cloudprovider/generic/vmcluster.py", line 297, in __init__
    super().__init__(**kwargs, security=self.security)
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/distributed/deploy/spec.py", line 275, in __init__
    super().__init__(
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/distributed/deploy/cluster.py", line 70, in __init__
    self._loop_runner = LoopRunner(loop=loop, asynchronous=asynchronous)
  File "/home/alexwalterbos/repos/magnet.me/services/recommender/similarity/venv/lib64/python3.10/site-packages/distributed/utils.py", line 533, in __init__
    raise RuntimeError(
RuntimeError: Constructing LoopRunner(loop=loop) without a running loop is not supported

Environment:

• Dask version: '2023.12.1'
• Python version: 3.10.13
• Operating System: Fedora Linux 6.6.8-200.fc39.x86_64
• Install method (conda, pip, source): pip

Thanks in advance for any assistance you can provide.