Feature: Typed hyperparmeters with default values

[joel-odlund]

Having hyperparameters in dictionaries is flexible, but it also limiting and error prone. It is hard to get an overview of which hyperparameters are available for a model, their names and what types they have. Configuring training runs is also error prone, as hyperparameter names must be referred to by strings, and cannot be validated by an IDE, or by an external configuration system. Scikitl-learn uses constructor arguments, which has its own problems, but it does give a natural way of declaring hyperparameters with type and default values, which Neuraxle is missing.

One solution to this might be to use dataclasses for hyperparameters. Dataclasses provide an ideomatic and compact way of defining data as python fields, with type information and default values built in. Using dataclasses for parameters would make it possible to validate hyperparameters before training and give meaningful error messages on errors. It would also make implementing models easier, as all available hyperparameters would be available in one place. There would be no need to use string indexing, and an IDE would be able to autocomplete hyperparameter names when coding.

For context, im currently implementing an ML experimentation environment with Neuraxle as the pipeline abstraction, and with an external configuration system inspired by Spacy 3.0. This issue comes up when trying to glue Neuraxle into a the external configuration.

[guillaume-chevalier]

Thank you for the suggestions @joel-odlund! The idea is interesting, however I'd like to know more. Here is what I think of your ideas, as well as some questions or comments:

• "cannot be validated by an IDE, or by an external configuration system": could you please give an idea of static validation of hyperparameters? Is that why dataclasses would be useful, so as to not forget to add some params?
• "validate hyperparameters before training": interesting idea! Like the global assertion wrappers we have.
• "possibly using dataclasses for hyperparameters": I feel this may become heavy. I would prefer a custom solution where the _HasHyperparamsMixin could allow the class to define its own hyperparameter requirements as well as the assertion wrapper.
• "It would also make implementing models easier, as all available hyperparameters would be available in one place": please note that this is possible with the getters and setters right now, such as pipeline.get_hyperparams().to_flat_dict() or something like this.
• "There would be no need to use string indexing": could you elaborate more on why it would be like this? I'm not sure to understand.

I would be curious to know what is your "external configuration system inspired by Spacy 3.0".

[joel-odlund]

Hi again.

• "cannot be validated by an IDE, or by an external configuration system": could you please give an idea of static validation of hyperparameters? Is that why dataclasses would be useful, so as to not forget to add some params? Yes, it would be helpful to have a static check that hyperparameters are complete and of correct type. For example, if some hyperparam is used late in the training code, or if the training is executed elsewhere, perhaps on a queue, it would take a long time to catch a missing parameter or a spelling error. Dataclasses, or some other typed construct, would make it possible to catch these bugs early.

• "validate hyperparameters before training": interesting idea! Like the global assertion wrappers we have. Yes, that would be nice. Ideally it would be supported by design, and not require any explicit assertions by the user.

• "possibly using dataclasses for hyperparameters": I feel this may become heavy. I would prefer a custom solution where the HasHyperparamsMixin could allow the class to define its own hyperparameter requirements as well as the assertion wrapper. I agree that dataclasses is somewhat heavy. In particular, it requires you to create a class with a name, and somehow associate it with the step. However, it allows you to compactly representing all hyperparams, with names, types and default values in a compact way, and checks could be implemented by the framework and not by the user. Im not sure what you have in mind for an assertion wrapper, but i do believe that asserting values / types could be done on a language level rather than on the pipeline level (ie by using typed values).

• "It would also make implementing models easier, as all available hyperparameters would be available in one place": please note that this is possible with the getters and setters right now, such as pipeline.get_hyperparams().to_flat_dict() or something like this Yes. I was referring to readability of the code when developing models and pipelines, not at runtime.

• "There would be no need to use string indexing": could you elaborate more on why it would be like this? I'm not sure to understand. A toy example: we use string indexing to access params in a dict

my_value = 1/self.params['alpha']

If params is a dataclass, the params are fields

my_value = 1/self.params.alpha

Here an IDE would autocomplete values, catch errors etc.

I would be curious to know what is your "external configuration system inspired by Spacy 3.0".

Im working in an NLP team and im working on a way to manage training and experimentation in an organized way. One thing that makes sense to me is to separate the implementation of models , components and pipelines from their particular configurations (ie hyperparameters). So we want to train a large number of versions of the same pipeline, with different parameters, without writing new code for each instace. Hence, we want an external configuration system. While working on this, Spacy released 3.0, with the main feature being an external configuration system. It is both compact and expressive, which is nice. It supports parameter validation by using function parameters. An annotated function creates your component, and parameters are represented as parameters to this function. I dont agree with this solution entirely, as it ties hyperparameters ti instantiation, much like scikitlearn. Spacy does not support hyperparameter optimization, which is where this design becomes problematic.

We are not ready to commit to Spacy as our framework, as it is not general enough, does not support hyperparam search, and we have non-nlp use cases. but we are borrowing the configuration system and combining it with neuraxle, to get the best of both worlds. I think a good config system is essential, and maybe adding one to Neuraxle could be a great thing!

[guillaume-chevalier]

@joel-odlund thanks for your answer! I logged the issue for the validation in #482.

Having configs is something I have long thought about. Have you seen issue #91? Would this be a suitable way to do configs? Configs would be validated the same way it'd be done with #482.

I would love to see you elaborate more with a code example of the ideas you'd like to see implemented.

Also please note that you and your company can contribute to Neuraxle. It's as simple as opening a pull request, and then signing the CLA following our procedure that can be seen here. We'd love to see your contributions as long as they are:

• Coded cleanly according to good coding standards,
• Do not break the API too much.

Thank you!

[guillaume-chevalier]

Hi @joel-odlund !

Thanks again for your comments. After thinking about it again, I've logged the issue #494 making use of more elegant code following your idea.

Don't hesitate to share more ideas like this one! :smiley:

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in the next 180 days. Thank you for your contributions.