Confidence level in augment.lm

[julian-urbano]

Hi!

I was checking augment to compute intervals, and wanted to use different confidence levels. I presumed this would be done via something like conf.level=0.9, thinking about how predict is used. However, the intervals are always the same, regardless of the confidence level.

I looked at the code for predict.lm and found this line here https://github.com/tidymodels/broom/blob/8bd38356c2f675f6889df2627d687d110a4c0fe7/R/stats-lm-tidiers.R#L136

I think it should read as follows

df <- augment_newdata(x, data, newdata, se_fit, interval, ...)

to forward other arguments, such as conf.level in this case.

Thoughts?

[simonpcouch]

This sounds like a good idea to me!

re: the comment be *incredibly* careful that the ... are passed correctly in the augment_newdata comments, checking in—does this sound good to you, @alexpghayes?

[julian-urbano]

Just wanted to nudge this a bit. I'm updating my course materials one year after and this is unfortunately not correct yet.

[petebaker]

Hi @julian-urbano, @simonpcouch and @alexpghayes

I would also like to nudge this topic along but I'm not sure about the interest level. I think that broom is fantastic and it's a great step in the right direction. However, being able to set the level for a CI but not a PI is a deal breaker for me and I believe it is a trap for the unwary.

Recently, I highly recommended broom to colleagues but once I found out the state of play w.r.t. prediction intervals I've highly un-recommended it. Here's why.

My colleagues wanted to study various prediction intervals 80%, 90% etc but they pointed out that the prediction intervals are always 95% which makes augment.lm essentially useless for them, and unfortunately they couldn't work out how to make it work so they abandoned augment.lm and rolled their own using predict.lm and level.

The documentation around ... is somewhat messy but the important problem is that ... is not implemented. The documentation gives a heads up that things can go wrong when options are chosen via ... which is perfectly sensible. (However, given that ... isn't implemented for prediction, then it is simply wrong). Am I missing something here?

I think broom is great but if standard options/arguments like level for predict.lm simply aren't implemented when it appears that they are available, then this seems dangerous to me.

If maintainers are interested, then I will have a go at a pull request.

Comments most welcome!

Cheers Peter

PS: Having produced both CIs and PIs for about 40 years now, then I would say that prediction intervals for new observations are often more useful than confidence intervals for the mean response (unless fixed at some arbitrary level of course).

[alexpghayes]

A PR adding a conf.level argument eventually passed to level would be welcome. I would not pass ... though, or at least I'd want to think a little more carefully if passing the dots might lead to some edge case breakages.

On Tue, Apr 12, 2022, 1:06 AM Peter Baker @.***> wrote:

Hi @julian-urbano https://github.com/julian-urbano, @simonpcouch https://github.com/simonpcouch and @alexpghayes https://github.com/alexpghayes

I would also like to nudge this topic along but I'm not sure about the interest level. I think that broom is fantastic and it's a great step in the right direction. However, being able to set the level for a CI but not a PI is a deal breaker for me and I believe it is a trap for the unwary.

Recently, I highly recommended broom to colleagues but once I found out the state of play w.r.t. prediction intervals I've highly un-recommended it. Here's why.

My colleagues wanted to study various prediction intervals 80%, 90% etc but they pointed out that the prediction intervals are always 95% which makes augment.lm essentially useless for them, and unfortunately they couldn't work out how to make it work so they abandoned augment.lm and rolled their own using predict.lm and level.

The documentation around ... is somewhat messy but the important problem is that ... is not implemented. The documentation gives a heads up that things can go wrong when options are chosen via ... which is perfectly sensible. (However, given that ... isn't implemented for prediction, then it is simply wrong). Am I missing something here?

I think broom is great but if standard options/arguments like level for predict.lm simply aren't implemented when it appears that they are available, then this seems dangerous to me.

If maintainers are interested, then I will have a go at a pull request.

Comments most welcome!

Cheers Peter

PS: Having produced both CIs and PIs for about 40 years now, then I would say that prediction intervals for new observations are often more useful than confidence intervals for the mean response (unless fixed at some arbitrary level of course).

— Reply to this email directly, view it on GitHub https://github.com/tidymodels/broom/issues/949#issuecomment-1096126978, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTBG26XT2N2BM4THQSGHILVEUHGHANCNFSM4SL43MOQ . You are receiving this because you were mentioned.Message ID: @.***>

[ptoche]

This would definitely be a nice addition. I think most people use tidy() to set confidence levels, rather than augment(), but I too tried it and noticed the confidence intervals weren't updated. Feels like a natural thing to try. Would be nice to have.

[julian-urbano]

So another year has passed, and I decided to have a go at this and create a pull request, as suggested by @alexpghayes. However, when testing I get this error:

Failure (test-stats-lm.R:9:3): lm tidier arguments
length(not_allowed) == 0 is not TRUE

`actual`:   FALSE
`expected`: TRUE 
Arguments level to `augment.lm` must be listed in the argument glossary.
Backtrace:
 1. modeltests::check_arguments(augment.lm)
      at test-stats-lm.R:9:2
 2. testthat::expect_true(...)

which seems to refer to the glossary on acceptable argument names. It looks like augment does not currently accept anything related to confidence levels.

Does this mean that this issue will not be addressed anytime soon? How about forwarding ... to predict?

[alexpghayes]

You'll need to make a second PR to alexpghayes/modeltests to add any new argument names to individual augment() methods.