Unified API for different ERM-based approaches

[jumutc]

We need a generic API suitable for different ERM-based approaches. Right now some signatures (for instance optimize method) are completely tailored towards only one hyperparameter λ. In general there should be enough abstraction in API to handle tons of existing approaches without unveiling the implementation details.

[BigCrunsh]

Hey @jumutc. I agree that the API is probably not perfect and has to be adapted. However, I am a fan of starting with something simple and make it more complex as soon as it becomes necessary. Indeed having the hyperparameter(s) as an explicit argument in optimize is a needless limitation. The idea was to simplify tuning via crossvalidation and grid search (see, e.g., https://github.com/JuliaStats/MLBase.jl/). Conceptually, I would argue that the hyperparameter belongs to the regularizer and should thus be a part of the method RegERM:

function optimize(method::RegERM; optimizer::Symbol=:l_bfgs)

Does that help? Do you have any other suggestions? I think it would be helpful, if you give some specific examples that are not captured by the current API to see which abstraction is missing.

[jumutc]

Hey @BigCrunsh ... One major thing to be considered (from my perspective) is how we can elaborate the easiest way to use such method-dependant interfaces and models for non-ML practitioner!? If one has to take care of cross-validation and fine-tuning of these hyperparameters it makes a package much harder to start with! Maybe we can couple RegERMs.jl with MLBase.jl somehow and alleviate practical usage of the first one?

P.S. Interface that you suggested is perfectly fine, but the instantiation of such RegERM method might be a trouble as I already mention above!