Configuring hyperparameter space for Auto* Models

[yarnabrina]

Description

As of now, users can pass a hyperparameter space using config argument which overrides the existing parameter grid. A new option update_config can be added so that users can make minor modifications in the default space.

Use case

Quite frequently the default hyperparameter space is good enough, but sometimes users may want to control it in a minor way. For example, user may want to fix the context length to a specific value or may want to restrict the number of layers. As of now, passing a dictionary corresponding to only that field is behaving as if that's the entire search space along with default values of other parameters, but users intention is to restrict only few specific dimensions in the default space.

[elephaint]

~To do what you want, you can access the default_config attribute of the Auto* models, and make changes:~ [deleted]

Update: the suggested solution doesn't work. Sorry for the inconvenience.

[yarnabrina]

Thank you.

For a interactive use, I think that is fine. Currently I am using these in automated pipelines through the sktime wrapper I added a few weeks back, where the model object itself is not exposed publicly to end users. So an ability to configure itself during initialization will be easier I feel.

This is the relevant adapter PR, if you want to take a look at the flow: https://github.com/sktime/sktime/pull/5962.

[elephaint]

Do you have an example of such an automated pipeline? The linked PR doesn't talk about Auto* models and I don't see an example piece of code of an automated pipeline in that PR.

[yarnabrina]

I have, but not readily as a public code but only for my office work.

The linked PR only introduces the wrapper, and users/developers are encouraged to define/contribute their own classes inheriting from it for different other models from neuralforecast.

This is a sample code for the general use of interfaced models:

# creating model instance configuring the hyperparameters
model = NeuralForecastRNN(freq="A-DEC", futr_exog_list=["ARMED", "POP"], max_steps=5)

# fitting the model
model.fit(y_train, X=X_train, fh=[1, 2, 3, 4])

# getting point predictions
model.predict(X=X_test)

Now, if it was NeuralForecastAutoRNN, as a user I may like to fix context length at 14, based on may be some specific data domain. I want to still search for best combinations of other hyperparameters, so I'd like to use:

NeuralforecastAutoRNN(update_config={"context_size": 14})
# or tune.choice([14]) if 14 causes any issues

As of now, users have to know that there's a private attribute _forecaster which they can configure themselves later. We can make it public, but I believe it goes against sklearn/sktime principle to modify public properties after definition, as model.config will no longer be in sync with model._forecaster.default_config.

[elephaint]

Thanks - Weighing the pros and cons for now we will update the docs to demonstrate the ability to access the attribute to attain the default config.

For the sktime wrapper I'd also suggest to indicate it in the docs; it's fairly easy and the additional complexity with adding a potentially conflicting additional input (for example, what happens if update_config and config are both specified?) is something we would like to avoid.

[yarnabrina]

Sure, makes sense. I'll create a specific sktime adapter for Auto* models documenting this plus a feature to extract best configuration depending on ray or optuna.

what happens if update_config and config are both specified

Just a suggestion for this, you can consider adding a validation check if both are non-empty and inform users that they are mutually exclusive.

I am considering to add this in the sktime adapter itself, but it may be difficult for us to validate if the provided keys in update_config are compatible with the selected Auto* models. So sometimes it may send some hyperparameters invalid for a model and that may lead to some surprising errors (I didn't check yet, so don't know).

[pranavvp16]

updating the default config didn't worked for me I tried doing same mentioned above by @elephaint, accessing the dictionary keys and updating the values for AutoLSTM. Opening an issue regarding it

[elephaint]

updating the default config didn't worked for me I tried doing same mentioned above by @elephaint, accessing the dictionary keys and updating the values for AutoLSTM. Opening an issue regarding it

~When updating the docs we found that it requires a deepcopy to properly copy the config. See the updated comment above (I know this doesn't solve the issue)~

Update: copying the default parameters turns out to be a bit more involved due to the way we've coded the default_config.