Disable shuffle for custom CV

[jmrichardson]

#658

[ghost]

All CLA requirements met.

[liususan091219]

Hi @jmrichardson, also it would be great if you could add a test for your pull request. That is, adding the two following two lines at the end of test_object in https://github.com/microsoft/FLAML/blob/main/test/automl/test_split.py#L176:

automl_settings["split_type"] = your custom split type
automl.fit(X, y, **automl_settings)

Thanks in advance!

[jmrichardson]

Hi @liususan091219

I would love to add the custom cv for test but unfortunately I am using a custom splitter from a package (mlfinlab) for which the code is copyrighted.

[sonichi]

Hi @liususan091219

I would love to add the custom cv for test but unfortunately I am using a custom splitter from a package (mlfinlab) for which the code is copyrighted.

Or simply use the current TestKFold(5) object and set shuffle=True. Like:

kf = TestKFold(5)
kf.shuffle = True
automl_settings["split_type"] = kf

[sonichi]

@jmrichardson Please let us know whether you would like to make the suggested changes in this PR. If so, we'll wait for that. If not, could you please create issues corresponding to the suggestions? Thanks.

[jmrichardson]

Hi, yes, I will make the update to add the test. I will try resubmitting the PR shortly and hopefully it updates this one instead of creating a new one.

[jmrichardson]

Hi, I updated the PR with suggestion for test. The only thing is that I couldn't figure out a way to verify that shuffle did indeed happen with the data. I couldn't find a "shuffle" parameter in automl for which to test if was true or not. Not sure if this is something that you would like to assert or not.

[sonichi]

Sorry, one more thing to fix before we can merge this PR: https://github.com/microsoft/FLAML/runs/7680527800?check_suite_focus=true#step:8:12

An easy way to avoid the lint issues is to use pre-commit.