General framework including problems other than regression problems

[lendle]

#3 and #4 are about regression problems specifically, but risk minimization is much more general. For example, density estimation can be formulated as minimizing the negative log density over a class of densities. We could consider a framework that includes both regression and non-regression problems. (I don't necessarily thing that we should do this, but it's worth talking about now since it will be much easier to start with such a framework at this stage instead of adding in later.)

We might have a type hierarchy like

abstract AbstractLoss
abstract RegressionLoss <: AbstractLoss
type DensityLoss <: AbstractLoss end
#others... <: AbstractLoss

where subtypes of RegressionLoss are those discussed in #3.

Would something like this be useful?

[BigCrunsh]

My intuition is that everything that can be expressed as a reg. empir. risk can / should be included. So I like the idea to add also non-regression methods. However, I think that the distinction should be made between different models rather than between the loss functions as long as it is not necessary.

[BigCrunsh]

Hm. Perhaps I am convinced that it makes sense to have such a distinction. @lendle, @lindahua: What do think about this?

[lendle]

I like the DiscriminativeLoss/potential GenerativeLoss names. If it turns out that we don't want to support generative models it will be easy to remove.

[BigCrunsh]

Agree, currently I don't see an application of GenerativeLoss, since the loss always compare a prediction with a ground-truth. Can I close that issue and discuss it again, when we have an implementation of a density estimation approach (e.g., http://jmlr.org/proceedings/papers/v15/mahapatruni11a/mahapatruni11a.pdf)?

[lindahua]

What I would suggest right now is to have:

abstract AbstractLoss
abstract RegressionLoss <: AbstractLoss

This leaves the opportunity to have other kinds of losses, but now we can focus on regression problems.