Roadmap.jl
Roadmap.jl copied to clipboard
JuliaStats
New stab at an interface for parametric models
Following-up on @nalimilan's suggestion in METADATA.jl/pull/4491, I am opening this issue to discuss an interface design I started implementing in ParametricModels.jl. A relevant issue is #4 about a general type hierarchy for statistical models.
In ParametricModels.jl, a model::ParametricModel promises to implement three functions: calibrate!(model,parameter), simulate(model,T::Int) and loglikelihood(model,data::Array). Then the package implements sugar for simulate(model,T::Int,n:Int) and simulate(model,Tn::{Int,1}) and loglikelihood(model,data::Array{Array,1}) so that you can easily work with different shapes of data (1 time series, 1 panel data of n iid individual and T observations each, n individuals with T_i observations each, ...). The package also implements a generic numerical optimization approach to compute the mle by wrapping likelihood() with Optim.jl's functions in mle().
While morally I clearly have in mind ParametricModel <: StatisticalModel, I definitely don't want to impose confint, vcov etc. I am also reserved about coef as I would like to leave the door open to:
calibrate!(model,parameter::ParametrizationA), calibrate!(model,parameter::ParametrizationB) etc. For instance in HiddenMarkovModels.jl there is a parametrization in terms of the transition matrices (dispatched on ::Tuple) and a parametrization in a space more agreeable to numerical optimization (the usual logit reparametrization).
Looking back at other suggestions made in #4, I personally don't think all ParametricModel have to offer a fit() method as you may want to use a model purely for simulation purposes, or on the contrary you might have several fitting methods in mind (like mle() and gmm()).
I look forward to receiving feedback on ParametricModels.jl.
While morally I clearly have in mind ParametricModel <: StatisticalModel, I definitely don't want to impose confint, vcov etc.
You can easily skip these methods if you don't want to provide an implementation. Currently, they raise an error in that case. The point is rather that it makes it possible to implement these if you want without creating function conflicts between packages, and to unify the semantics.
I am also reserved about coef as I would like to leave the door open to: calibrate!(model,parameter::ParametrizationA), calibrate!(model,parameter::ParametrizationB) etc. For instance in HiddenMarkovModels.jl there is a parametrization in terms of the transition matrices (dispatched on ::Tuple) and a parametrization in a space more agreeable to numerical optimization (the usual logit reparametrization).
I don't really understand the problem here. Are you saying that coef could return two different sets of coefficients, one for each parameterization? If so, then another coef(model, parameterization) method could be added, and coef would return the default one.
Looking back at other suggestions made in #4, I personally don't think all ParametricModel have to offer a fit() method as you may want to use a model purely for simulation purposes, or on the contrary you might have several fitting methods in mind (like mle() and gmm()).
Again, I don't see this as an issue. fit can take arguments to specify which optimization method should be used. This is actually what GLM.jl does already AFAIK.
I'd like to ask two questions. Is this specifically for dynamical models/time series? And what do you mean by "model calibration" or what is the intendet use of calibrate! in this context? From the code it looks like a assignment operation which is supposed to replace or extend the type constructor.
@nalimilan Thanks for the comments. I am generally fine with all of them. In fact I didn't say there was an issue or problem to start with :).
For my work it is convenient to have a notion of "parametric model" as described above. If you can skip an interface's methods at your leisure, I am more than happy to declare ParametricModel <: StatisticalModel. Then it seems like we are more talking about semantic guidelines, which I actually think make a lot of sense. Making sure the Julia stats ecosystem uses coef() rather than param(), and fit(model,:mle) across the board or whatever.
So to sum up are your recommending that I implement a fit(model,:mle) and a default coef(model) where it makes sense in my packages or is there something else?
@mschauer This is not specifically for time series. I am just trying to come up with a common interface fit(model,data) and simulate(model,datashape) for single time series (iid being a special case), panel data, etc.
calibrate!() is the evaluation of the model at a given parameter value. A model is a mapping from parameters to distributions, calibrate!() is the evaluation of this mapping at a particular parameter. Then simulate is a draw from the calibrated model. calibrate! would be called "solving the model" in some areas of economics and can involve a lot of work. Maybe HiddenMarkovModels.jl provides a good example?
Is calibrate! the same as fit!? If not, how do they differ? It would be best to unify the verbs as much as possible.
Ah, statistical babylon, I and maybe others expected model calibration to involve the data.
@tbreloff Nope. calibrate!(normal,(mu,sigma)) would say here is normal with mean mu and sd sigma, calibrate!(us-economy-model,agent-preferences) would say here is how would behave the US economy if agents had this type of preferences in my model.
fit(model,data) goes in the other direction. fit(normal,100 observations) says here is the method of moment or mle estimate of the mean and the sd from the data. fit(us-economy-model,5 years of data) says here is the distribution of preferences in the US population according to my model and observations after the crisis.
@mschauer Ah, I see, well I am happy to rename calibrate!(). Do people have suggestions? evaluate!()?
I see... and I guess the verb makes sense, it just also sounds similar to fit. Here's some other possible verbs, but maybe calibrate is the best:
- align
- balance
- finetune
- readjust
ps- I don't like evaluate... that sounds too similar to simulate
Thinking about this another way, does it really make sense to "re-calibrate" the model? What if you were to do: composed_model = compose(us-economy-model, agent-preferences) and create a new object that contains the calibrated model. Then this discussion would be moot.
In my view a model does not need to be calibrated when initiated. The model is the mapping from parameter to distributions and you don't necessarily want to distinguish a specific parameter value ex-ante. Maybe all you want to do is to fit the model from data and you will never calibrate it. In this sense calibrate! is not always a re-calibration.
And let me add: this is also why coef() is not necessarily defined for a model, in my view.
...this might or might not be a good idea, but if this is what we want then we'll have to have a model type and a fitted model type where the latter is also a function of the data. We should consider it the extra complexity added is outweighted by the usefulness of having a model without data. It might be really cool with this sepration or it might be really cumbersome to work with. I don't know yet.
we'll have to have a model type and a fitted model type where the latter is also a function of the data.
It would make it trivial to refit the same model to a different dataset, as is necessary with cross-validation and ensemble methods. scikit-learn has something similar, but it doesn't separate the model specification from the learned coefficients; instead it automatically assumes that the model's constructor's arguments are the model parameters. It's a bit wonky.
For me data-less models are an absolute must-have. There is the model on one side and the data on the other side. "Fitting", "estimating" ("calibrating" in some circles :) ) is bringing the model and the data together.
I am not sure about the distinction between a Model and a FittedModel. What about a Normal(mu,sigma) taken at (mu,sigma)=(2.3,.7)? There is no data but you can simulate from it, it can return coef() etc. (This is what I was calling a "calibrated model" for lack of a better word).
Also I don't see why a FittedModel should carry its data with it.
I think I'd call the "calibrated model" a probablity distribution and make it a subtype of Distribution.
In practice, I think it's quite convenient to carry a reference to the data that was used for fitting the model. It's how all out present "model objects" work right now and how all such model object in R I know of works. It might not be necessary, but I think it is convenient.
What about a Normal(mu,sigma) taken at (mu,sigma)=(2.3,.7)? There is no data but you can simulate from it, it can return coef() etc. (This is what I was calling a "calibrated model" for lack of a better word).
I think this is the perspective I'm hung up on. From my perspective:
-
Normal(mu, sigma)with(mu,sigma)=(2.3,.7)is a model. - You can sample from that model
- You may or may want to fit the model parameters to empirical data
- One would only simulate a model which generates a dependent path (ARIMA for example). Each data-point would be a sample.
- One could compose multiple models together to create some meta-model (
us-economy-modelandagent-preferencesfor example)
I'm sure I am missing something... thoughts?
@tbreloff Typically, a model is a family of probability distributions indexed by the parameters of the distributions so as mentioned above, I'd call Normal(μ, σ) with (μ,σ)=(2.3,.7) a distribution and not model. If μ and σ were left unspecified, I'd call it a model.
I'd also prefer to use another term than calibrate, as it's used to mean something completely different in other contexts. An example from Harrell's Regression Modeling Strategies:
Here we concern ourselves with the reliability or calibration of a model, meaning the ability of the model to predict future observations as well as it appeared to predict the responses at hand.
@BenConnault Isn't calibrate what is called predict in R and in GLM.jl, i.e. given a set of model parameters and the values of independent/explanatory variables, estimate the expected outcome?
@andreasnoack This would mean the calibrated model has forgotten where it comes from and N(mu,1) taken at mu=1 would be the same as N(1,sigma) taken at sigma=1. As a consequence you would not be able to compute things for which you need to move in parameter space, such as the Fisher information matrix.
@nalimilan I dislike calibrate!() too but I am having a hard time coming up with a better term. Maybe specify!()?
It is definitely not the same as predict though. It is what you do when you write f(theta) for a parametrized density but can involve intense computations as in the specify!(us-economy-model,agent-preferences) example.
So to sum up are your recommending that I implement a fit(model,:mle) and a default coef(model) where it makes sense in my packages or is there something else?
@BenConnault Yes, more or less. Though I would write fit(model, MLE) with MLE a type, or something similar, so that the method and the model type are specialized on the fitting method for efficiency. Again, have a look at GLM.jl.
I don't understand your two last comments TBH. Looks like we have very different ways of presenting things. Do you think you could give a small example of a model with simple functions, and what it would mean to "calibrate" that model?
I'd call Normal(μ, σ) with (μ,σ)=(2.3,.7) a distribution and not model.
I get what you mean, but in some sense: Distribution <: Model The important difference in terminology (I think) is that a distribution is generative, meaning you can sample from it. So in this case I agree Normal should be called a distribution, but that model is a more abstract concept.
If μ and σ were left unspecified, I'd call it a model
I might call this a "model form" or "model type"? It seems similar to comparing the julia type Normal to the julia object Normal(2.3, 0.7). One is a type, and one is an instance.
@tbreloff I think @andreasnoack's point is that a model is something specifying the relationships between variables, but with coefficients which can vary. Then you have to estimate these coefficients (or you can give them arbitrary values) to be able to generate data.
@BenConnault From that example, am I right understanding that fitting the model is a way to estimate the values of the theta coefficients, and that calibrating the model means attributing an arbitrary (based on a priori hypotheses, or whatever) value to theta? And that in both cases these theta values allow you to simulate data from the model?
@nalimilan exactly! or almost exactly. I would personally favor fit!(model,data) = calibrate!(model,estimate(model,data)) as I would de-emphasize calibrating the model at estimated values.
erf @nalimilan sorry I think that might be what you were saying. I just don't use "fit" usually and I was more under the impression that people often use fit = estimate + calibrate.
@BenConnault From your posts, I get the feeling that, from your perspective, calibrate is the same thing as construct. i.e. There's no need for a verb... just a type constructor.
OK, so now I think I see your point. Calibrating can be seen as bypassing the estimation to define the coefficients. I could imagine calling this operation setcoefs!, but it would be nice to use a term from the relevant literature. Unfortunately, calibration really looks ambiguous, as even in the field of economics some take it as meaning "estimation of the parameters given a specific functional form":
http://stats.stackexchange.com/questions/27672/example-of-estimation-vs-calibration
(many other Google results imply the same meaning)
Any other possibilities?
@tbreloff it is definitely true that by calibrate!(model,parameter) I pretty much mean model(parameter) as in Normal(1) but I don't think I can make the constructor work for more complicated models, dispatching, inheritance, etc. Maybe it's just my limited Julia skills, I don't know. I would have to think about it more.
@nalimilan yes I agree calibrate!() sucks but unfortunately I haven't found the perfect term despite being familiar with the relevant literature. I like setcoefs!() much better already. I was saying specify!() earlier but I think I prefer setcoefs!().