evalml icon indicating copy to clipboard operation
evalml copied to clipboard

DaskEngine: Repeated Use with Context Manager

Open chukarsten opened this issue 3 years ago • 1 comments

Per #2667 , it was noted that a DaskEngine can't be used in a context manager as the manager will shutdown the associated cluster. We should probably modify the __enter__() function to restart the cluster if it's shutdown.

def test_automl_dask_called_twice(
    X_y_binary_cls,
):
    """Making sure that the max_iterations parameter limits the number of pipelines run."""

    X, y = X_y_binary_cls
    max_iterations = 4
    engine = DaskEngine()
    with engine:
        par_automl = AutoMLSearch(
            X_train=X,
            y_train=y,
            problem_type="binary",
            engine=engine,
            max_iterations=max_iterations,
        )
        par_automl.search()

    with engine:
        par_automl = AutoMLSearch(
            X_train=X,
            y_train=y,
            problem_type="binary",
            engine=engine,
            max_iterations=max_iterations,
        )
        par_automl.search()

The above code snippet should be modified into a test that passes.

chukarsten avatar Sep 01 '21 00:09 chukarsten

@chukarsten what's the priority on this?

dsherry avatar Sep 01 '21 19:09 dsherry