projpred
projpred copied to clipboard
stan-dev
Omitting intercept in submodel(s) does not work
It seems like setting argument intercept = FALSE in project() has no effect, as shown by the reprex below (tested on develop branch). The problem might be that intercept is not taken into account in these lines: https://github.com/stan-dev/projpred/blob/0cabd73548eec32e9b675f15cd39d377a40f180a/R/projfun.R#L84-L92 but the problem might also be that project_submodel() has an argument intercept which is not used, though.
In the reprex below, I illustrate the issue with project() but I think it also occurs when projecting in varsel() and cv_varsel().
library(projpred)
library(rstanarm)
options(mc.cores = parallel::detectCores(logical = FALSE))
data('df_gaussian', package = 'projpred')
mydat <- cbind("y" = df_gaussian$y, as.data.frame(df_gaussian$x))
myfit <- stan_glm(y ~ V1 + V2 + V3 + V4 + V5,
family = gaussian(),
data = mydat,
prior = hs(df = 1, global_scale=0.01), seed = 1140350788)
myproj <- project(myfit,
solution_terms = c("V3", "V2"),
ndraws = 20,
intercept = FALSE)
colnames(as.matrix(myproj))
## --> Gives:
## [1] "Intercept" "V3" "V2" "sigma"
When discussing this, we got to the conclusion that we would not allow projections without intercept. Therefore, I'm removing this argument altogether, as we don't think it makes sense to compute projections without intercept in general. We left the intercept argument for legacy reasons but it's true that then inside we force it to be TRUE for the projections.
Thanks for addressing this in commit 55cf78bc81ae92c1fc9dd041b67bead231552bfc. As far as I understood that commit, you're using refmodel$intercept as a guidance on whether to include the intercept in the submodels or not. This is a good solution, I think. (In contrast, forcing the submodels to always include an intercept would probably have been less desirable, as there may be cases like this one where excluding the intercept from the submodels indeed makes sense. But I agree that these are probably the same cases where the intercept would also be excluded from the reference model, so using refmodel$intercept as a guidance should be fine.)
However, if I now run the reprex from above but exclude the intercept from the reference model, then the submodel still contains an intercept (and that intercept is non-zero):
library(projpred)
library(rstanarm)
options(mc.cores = parallel::detectCores(logical = FALSE))
data('df_gaussian', package = 'projpred')
mydat <- cbind("y" = df_gaussian$y, as.data.frame(df_gaussian$x))
myfit <- stan_glm(y ~ 0 + V1 + V2 + V3 + V4 + V5,
family = gaussian(),
data = mydat,
prior = hs(df = 1, global_scale=0.01), seed = 1140350788)
myproj <- project(myfit,
solution_terms = c("V3", "V2"),
ndraws = 20)
myprojmat <- as.matrix(myproj)
colnames(myprojmat)
head(myprojmat)
I have created PR #100 where I try to fix this.
The reason we didn't dedicate time to models without intercepts is that in general that's not a good practice, just commenting for reference. However, now that the PR is up we can probably do a bit more work to make sure that everything works as intended and merge it.
Yes, I agree that in general, no-intercept models are not a good practice. But as I said, there might be (very special) cases where it makes sense to omit the intercept, e.g. that case.
Sorry, it took me some time, but I think I understand now what you wanted to say with
When discussing this, we got to the conclusion that we would not allow projections without intercept.
I guess your point is that submodels without an intercept do not make sense. In other words, it makes sense to always project onto submodels which include an intercept, even if the reference model doesn't include one. The reason is that even if the true intercept in the reference model were zero, this only holds for the reference model. The submodels only approximate the reference model (in terms of KL divergence of the predictive distributions), so the submodels might require an intercept nonetheless. Thus, we can never be sure that the intercept in a submodel is zero, so it makes sense to always estimate it.
Was this your reasoning? If yes, then we could re-allow reference models without an intercept and always project onto submodels which include an intercept. PR #100 is then definitely closed (i.e., does not need to be re-opened).
Yes, that’s our reasoning.
From: Frank Weber @.> Date: Friday, 21. January 2022 at 12.07 To: stan-dev/projpred @.> Cc: Alejandro Catalina @.>, Comment @.> Subject: Re: [stan-dev/projpred] Omitting intercept in submodel(s) does not work (#96)
Sorry, it took me some time, but I think I understand now what you wanted to say with
When discussing this, we got to the conclusion that we would not allow projections without intercept.
I guess your point is that submodels without an intercept do not make sense. In other words, it makes sense to always project onto submodels which include an intercept, even if the reference model doesn't include one. The reason is that even if the true intercept in the reference model were zero, this only holds for the reference model. The submodels only approximate the reference model (in terms of KL divergence of the predictive distributions), so the submodels might require an intercept nonetheless. Thus, we can never be sure that the intercept in a submodel is zero, so it makes sense to always estimate it.
Was this your reasoning? If yes, then we could re-allow reference models without an intercept and always project onto submodels which include an intercept. PR #100https://github.com/stan-dev/projpred/pull/100 is then definitely closed (i.e., does not need to be re-opened).
— Reply to this email directly, view it on GitHubhttps://github.com/stan-dev/projpred/issues/96#issuecomment-1018361615, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABZ5FHZELDUWQHQNBJ7PPXLUXEV6LANCNFSM4ZOUOCFA. You are receiving this because you commented.Message ID: @.***>
Despite our last two comments here, I still have a question concerning this case mentioned several times before: If, in such a latent projection case, we have centered predictors and no multilevel or additive terms, then I still think it should be ok to fit the submodels without an intercept because their true intercept is zero by construction. Do you agree?
Nevertheless, I agree that at least for now, it should be ok to always include the intercept in the submodels (and thereby also allow reference models excluding an intercept), firstly because situations where the submodel intercept can be omitted (like the latent projection mentioned above) should not be too common (at least compared to situations where the submodel intercept is necessary) and secondly because including the submodel intercept where it is not needed shouldn't harm too much.
Yes I do agree but, as you mention, there is no harm in including the intercept in those cases, so it should be fine.
Get Outlook for iOShttps://aka.ms/o0ukef
From: Frank Weber @.> Sent: Saturday, March 12, 2022 9:57:32 PM To: stan-dev/projpred @.> Cc: Alejandro Catalina @.>; Comment @.> Subject: Re: [stan-dev/projpred] Omitting intercept in submodel(s) does not work (#96)
Despite our last two comments here, I still have a question concerning this casehttps://discourse.mc-stan.org/t/variable-selection-with-ordinal-model/17242 mentioned several times before: If, in such a latent projection case, we have centered predictors and no multilevel or additive terms, then I still think it should be ok to fit the submodels without an intercept because their true intercept is zero by construction. Do you agree?
Nevertheless, I agree that at least for now, it should be ok to always include the intercept in the submodels (and thereby also allow reference models excluding an intercept), firstly because situations where the submodel intercept can be omitted (like the latent projection mentioned above) should not be too common (at least compared to situations where the submodel intercept is necessary) and secondly because including the submodel intercept where it is not needed shouldn't harm too much.
— Reply to this email directly, view it on GitHubhttps://github.com/stan-dev/projpred/issues/96#issuecomment-1065953169, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABZ5FH2QOD6H3KCTZJY6G2DU7TZKZANCNFSM4ZOUOCFA. You are receiving this because you commented.Message ID: @.***>
Ok, thanks :)
Personally, I could imagine a minor loss of efficiency (in terms of precision in the estimation of the remaining parameters) if an intercept is included where it is not needed. That's why I wrote "shouldn't harm too much" above. But I might be wrong.
Yeah, that for sure, there would be a bit of correlation, but should be minor. We cam include an option for these cases but then again, should be uncommon case.
Get Outlook for iOShttps://aka.ms/o0ukef
From: Frank Weber @.> Sent: Saturday, March 12, 2022 10:04:50 PM To: stan-dev/projpred @.> Cc: Alejandro Catalina @.>; Comment @.> Subject: Re: [stan-dev/projpred] Omitting intercept in submodel(s) does not work (#96)
Ok, thanks :)
Personally, I could imagine a minor loss of efficiency (in terms of precision in the estimation of the remaining parameters) if an intercept is included where it is not needed. That's why I wrote "shouldn't harm too much" above. But I might be wrong.
— Reply to this email directly, view it on GitHubhttps://github.com/stan-dev/projpred/issues/96#issuecomment-1065954151, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABZ5FHY4TNOURHEABVNXJH3U7T2GFANCNFSM4ZOUOCFA. You are receiving this because you commented.Message ID: @.***>