dask-kubernetes icon indicating copy to clipboard operation
dask-kubernetes copied to clipboard

Experimental `KubeCluster` Adaptive mode is broken

Open BitTheByte opened this issue 2 years ago • 1 comments

I hate being the first issue after minutes of release, KubeCluster2 adapt is broken

from dask_kubernetes.experimental import KubeCluster
from dask.distributed import Client

cluster = KubeCluster(name='helloworld')
cluster.adapt(minimum=1, maximum=4)
client = Client(cluster)
2022-04-26 16:15:41,067 - distributed.deploy.adaptive - INFO - Adaptive scaling started: minimum=4 maximum=4
2022-04-26 16:15:42,086 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x00000260D5603490>>, <Task finished name='Task-39' coro=<AdaptiveCore.__init__.<locals>._adapt() done, defined at C:\Python39\lib\site-packages\distributed\deploy\adaptive_core.py:119> exception=AttributeError("'KubeCluster' object has no attribute 'workers'")>)
Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\tornado\ioloop.py", line 741, in _run_callback
    ret = callback()
  File "C:\Python39\lib\site-packages\tornado\ioloop.py", line 765, in _discard_future_result
    future.result()
  File "C:\Python39\lib\site-packages\distributed\deploy\adaptive_core.py", line 122, in _adapt
    await core.adapt()
  File "C:\Python39\lib\site-packages\distributed\deploy\adaptive_core.py", line 229, in adapt
    recommendations = await self.recommendations(target)
  File "C:\Python39\lib\site-packages\distributed\deploy\adaptive.py", line 154, in recommendations
    if len(self.plan) != len(self.requested):
  File "C:\Python39\lib\site-packages\distributed\deploy\adaptive.py", line 122, in plan
    return self.cluster.plan
  File "C:\Python39\lib\site-packages\distributed\deploy\cluster.py", line 505, in plan
    return set(self.workers)
AttributeError: 'KubeCluster' object has no attribute 'workers'
2022-04-26 16:15:43,076 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x00000260D5603490>>, <Task finished name='Task-43' coro=<AdaptiveCore.__init__.<locals>._adapt() done, defined at C:\Python39\lib\site-packages\distributed\deploy\adaptive_core.py:119> exception=AttributeError("'KubeCluster' object has no attribute 'workers'")>)

Dask, dask-kubernetes, distributed: 2022.04.1

BitTheByte avatar Apr 26 '22 14:04 BitTheByte

@BitTheByte Yes, we're aware of this issue. Adaptive mode has not been implemented yet for KubeCluster. We'll update the docs to reflect

Matt711 avatar Apr 26 '22 15:04 Matt711