stan-dev
Release CmdStan 2.37
Apologies for the late issue creation
Feature freeze June 5, 2025:
- [x] Ensure all expiring language deprecations have been removed or given new removal dates.
- [x] Create and merge version updating pull requests in Math/Stan/Cmdstan. These should be the last PRs accepted before the freeze.
- [x] Create Math/Stan RC releases.
- [x] Create Stanc3 RC binary.
- [x] Create a release candidate tarball for x86. Make sure RC tarballs include stanc3 binaries.
- [x] Check external links in docs (i.e. TBB docs link) - use https://github.com/tcort/markdown-link-check
- [x] Create a release candidate feature/bugfix list (major features/bugfixes that need testing, link to new docs in Github)
- [x] Run CmdStanR tests with the RC tarball.
- [x] Run CmdStanPy tests with the RC tarball.
- [x] Make a Discourse RC post.
- [ ] Post a tweet with a link to the Discourse RC post.
Release June 19?, 2025:
- [ ] Create the Math Release notes.
- [ ] Create the Stan Release notes.
- [ ] Create the Cmdstan Release notes.
- [ ] Create the Stanc3 Release notes, include new deprecations and removals.
- [ ] Rebuild and publish docs for the new version.
- [ ] Check that docs for the previous release links correctly to the newest docs.
- [ ] Create the Math release.
- [ ] Create the Stan release.
- [ ] Create the Stanc3 release.
- [ ] Create x86 CmdStan tarballs (check version, check that the extracted folder is in the cmdstan-version format).
- [ ] Create non-x86 CmdStan tarballs.
- [ ] Run CmdStanR tests with the release tarball.
- [ ] Make a Stan blog release announcement post (thank the sponsors and all contributors, mention new devs).
- [ ] Link to the blog post in a Discourse thread.
- [ ] Make a Twitter announcement.
Preliminary release notes as of this morning: https://gist.github.com/WardBrian/db30d2866976546ae1b05b8c056e4649
https://github.com/stan-dev/math/issues/3189 was reported this morning, but has been around for at least one release cycle at this point, so I'm not sure it's worth holding the RCs. We can merge it during the freeze if we find an acceptable fix.
Discourse post up: https://discourse.mc-stan.org/t/cmdstan-stan-2-37-release-candidate/39655 CmdStanPy tests running: https://github.com/stan-dev/cmdstanpy/actions/runs/15471110076/job/43555602926
The CmdStanPy tests passed, and @jgabry confirmed that cmdstanr looks good locally (the tarball check github action they have is pretty out of date)
A couple issues in the embedded Laplace approximation were spotted today (courtesy of @avehtari):
- https://github.com/stan-dev/stanc3/pull/1519
- https://github.com/stan-dev/math/pull/3197
These will be worthy of an RC2, though it probably makes sense to wait a day or two after they are merged to see if anything else comes in
There are several issues that have arisen that will likely be blockers for a release.
Laplace related:
- https://github.com/stan-dev/math/pull/3209 and https://github.com/stan-dev/stanc3/pull/1521
- https://github.com/stan-dev/math/pull/3210 and https://github.com/stan-dev/stanc3/pull/1523
- https://github.com/stan-dev/math/issues/3205
Other:
- https://github.com/stan-dev/math/pull/3221
- https://github.com/stan-dev/math/pull/3217
- https://github.com/stan-dev/stan/pull/3352
- Maybe https://github.com/stan-dev/stanc3/pull/1535
My personal goal is to get RC3 out this week. https://github.com/stan-dev/math/pull/3221 is pretty close to mergeable I think, and then it's just a question of what do we do in terms of improving laplace's numerics before this release versus after
cc @SteveBronder @serban-nicusor-toptal
Yes I think https://github.com/stan-dev/math/pull/3221 should be good in the next few days
For laplace, I have this branch that seems to fix the roach model
https://github.com/stan-dev/math/tree/fix/laplace-wolfe
I do not like the idea of releasing laplace and not having it work for these kinds of problems. I can try to get this all cleaned up this week. If I cannot then I think we should hide or remove laplace from this release.
It is now more than 8 months since the last release on 10 Dec 2024.
Maybe we should reconsider timed releases?
I'm also worried that Laplace is soaking up more of @SteveBronder's time than it's worth as a feature, with no end in sight. My impression of its value has been highly influenced by the string of examples that have come up since the first release candidate where it fails.
Is there a use case for the current implementation that's going to help our users fit models they couldn't otherwise fit? And can we describe what that use case is in a way they will understand? If not, I'm OK just tabling Laplace indefinitely. If so, then we should document those things and release and let people know every other use is experimental and they shouldn't try things like the "roach model", whatever that is (something from posteriordb or BUGS volumes?). We know this isn't going to work everywhere, so it's just a matter of where the rough edges are and how well we can describe them to users. My real fear is that users are going to try this and get super frustrated when it doesn't work because the boundaries haven't been explained to them. This is already a problem with HMC itself.
I'm not sure @charlesm93 is currently subscribed to this issue, but he's probably one of the better positioned people to answer your questions @bob-carpenter so I'm pinging him
I would loop @avehtari in the discussion as well.
I agree with @bob-carpenter we shouldn't freeze the release for too long and if this means holding off on releasing the embedded Laplace, then so be it.
I do think it is useful to be able to test this feature in the way we have been doing in the last few weeks and I want a branch with the embedded Laplace to be easily accessible.
We have examples where it works well:
- GP with a Poisson likelihood (disease map of Finland)
- Horseshoe linear regression on prostate cancer data
- GP with a Bernoulli likelihood (SKIM on prostate cancer data)
And examples where it struggles:
- GP motorcycle (known to be a hard problem)
- Roach (but isn't this solved with the Wolf linesearch?)
Are there other models on which the embedded Laplace has been tested?
I want a branch with the embedded Laplace to be easily accessible
We could (rather trivially) add a flag to the compiler which you need to pass to enable the signatures, something like stanc --laplace-beta-opt-in. Then it would be in the release, easy to test, but not without the user explicitly acknowledging the testing nature.
I would only agree to do this if we were confident that it would be necessary for one version only. If we think it wouldn't be in a generally available state by the end of this year, we'd be better off un-merging it IMO.
That being said, I also think it's pretty reasonable to proceed without this flag and just make users aware of the fact that it is an approximation that may not work on all their problems.
It's a judgement call I leave up to you, @SteveBronder, and @avehtari, but I'd appreciate if a decision was made either way this week
@SteveBronder and I are waiting @charlesm93 to wake up in Vancouver, so that we can have a meeting to discuss
Maybe we should reconsider timed releases?
I've been against timed releases since we suggested them years ago. I think a 3 month vibe check of "has enough happened that we should do a release" is nice, but I don't like the fervor and rush as people try to get in PRs before a timed release.
I'm also worried that Laplace is soaking up more of @SteveBronder's time than it's worth as a feature, with no end in sight. My impression of its value has been highly influenced by the string of examples that have come up since the first release candidate where it fails.
It's taken up some time but I think there is light at the end of the tunnel.
Is there a use case for the current implementation that's going to help our users fit models they couldn't otherwise fit? And can we describe what that use case is in a way they will understand? If not, I'm OK just tabling Laplace indefinitely. If so, then we should document those things and release and let people know every other use is experimental and they shouldn't try things like the "roach model", whatever that is (something from posteriordb or BUGS volumes?). We know this isn't going to work everywhere, so it's just a matter of where the rough edges are and how well we can describe them to users. My real fear is that users are going to try this and get super frustrated when it doesn't work because the boundaries haven't been explained to them. This is already a problem with HMC itself.
100% agree with this, especially the last line. In Laplace's current state I would not be happy with most users playing with the general Laplace. At the same time giving Aki access to the impl has been great for finding trouble spots. That's why I like @WardBrian's idea of adding the --laplace-beta-opt-in flag for this release.
We could (rather trivially) add a flag to the compiler which you need to pass to enable the signatures, something like
stanc --laplace-beta-opt-in. Then it would be in the release, easy to test, but not without the user explicitly acknowledging the testing nature.
I would only agree to do this if we were confident that it would be necessary for one version only. If we think it wouldn't be in a generally available state by the end of this year, we'd be better off un-merging it IMO.
The main piece left for Laplace is merging the wolfe line search code. If we want to do the release this week then idt I'll have time to get it in. As long as we have Laplace accessable to Aki and friends via the flag then I think this is a good approach for this release and next release we will have Laplace without the flag. This also gives us more time to reach our to other power users and have them test out Laplace
That being said, I also think it's pretty reasonable to proceed without this flag and just make users aware of the fact that it is an approximation that may not work on all their problems.
If we do not have the flag then I'd rather not expose Laplace at all this release. @avehtari and @charlesm93 and I are going to meet today or Thursday to go over Laplace and make sure we've caught any other edge cases we can think of.
"roach model", whatever that is (something from posteriordb or BUGS volumes?).
@bob-carpenter the roach dataset is from "Data Analysis Using Regression and Multilevel/Hierarchical Models" and "Regression and Other Stories". Aki has the R and Stan code below
https://github.com/stan-dev/math/issues/3205#issuecomment-3178811967
The initial values of the theta vector we estimate in Laplace is a vector of zeros by default. For this particular problem, a value of 0 for theta is pretty far away from the true value. So the newton step we are doing inside of Laplace had a huge gradient at 0 which then led the newton step to shoot off to crazy values. So I had to write a wolfe line search that for the first few values can tamper down the step size. Then, once theta gets to nicer values the line search increases it's step size (normally back to around 1)
The main piece left for Laplace is merging the wolfe line search code. If we want to do the release this week then idt I'll have time to get it in. As long as we have Laplace accessable to Aki and friends via the flag then I think this is a good approach for this release and next release we will have Laplace without the flag. This also gives us more time to reach our to other power users and have them test out Laplace
I may be misreading this, but if you're suggesting we release with laplace behind a flag but not with the latest version using the wolfe line search, I'm not sure I see the point compared to just not having it in the release. If testers needed a specific branch to actually get it to work, then we could just as well ask them to use a specific branch to get access to it at all.
I may be misreading this, but if you're suggesting we release with laplace behind a flag but not with the latest version using the wolfe line search, I'm not sure I see the point compared to just not having it in the release. If testers needed a specific branch to actually get it to work, then we could just as well ask them to use a specific branch to get access to it at all.
@WardBrian and I just spoke and what we will do instead of a flag is remove the laplace signatures from stanc3 for this release, then immediately put them back in. Then we have a full release cycle to test laplace out and time to merge the wolfe line search
I've prepared https://github.com/stan-dev/stanc3/pull/1539
In short, I think the summary is:
- If Wolfe is merged before the remaining necessary non-laplace bugfixes, we can add a beta flag but include all the code in the full release.
- If it isn't, and we are going to ask testers to use a branch anyway, the above PR should be merged which completely hides the laplace feature in the released code
With https://github.com/stan-dev/math/pull/3221 merged we should be good to run RC3 on Monday.
RC3 is out!
Hopefully we will be good to go on the full release next week.
At least one bug in RC3 which is not in RC2. Stan model https://github.com/avehtari/casestudies/blob/master/Birthdays/gpbf4.stan fails to run with an error
Chain 1 gpbf4: stan/lib/stan_math/lib/eigen_3.4.0/Eigen/src/Core/DenseBase.h:261: void Eigen::DenseBase<Derived>::resize(Eigen::Index, Eigen::Index) [with Derived = Eigen::Map<Eigen::Matrix<double, -1, 1>, 0, Eigen::Stride<0, 0> >; Eigen::Index = long int]: Assertion `rows == this->rows() && cols == this->cols() && "DenseBase::resize() does not actually allow to resize."' failed.
The model is part of case study https://users.aalto.fi/~ave/casestudies/Birthdays/birthdays.html. If needed, I can try to make a minimal example.
Thanks Aki. I’m guessing that was introduced by https://github.com/stan-dev/math/pull/3221 purely because it mentions Eigen Maps, but I’ll try to find the exact cause
Backtrace blames your diagSPD_periodic function calling log_modified_bessel_first_kind:
#15 stan::math::apply_scalar_binary<stan::math::log_modified_bessel_first_kind<std::vector<int, std::allocator<int> >, double&, (void*)0>(std::vector<int, std::allocator<int> >&&, double&)::{lambda(auto:1&&, auto:2&&)#1}, std::vector<int, std::allocator<int> >, double&, (void*)0, (void*)0>(std::vector<int, std::allocator<int> >&&, double&, double&) (f=..., x=..., y=@0x7fffffffb658: 1.8872158590898946) at stan/lib/stan_math/stan/math/prim/functor/apply_scalar_binary.hpp:311
#16 0x00005555555add22 in stan::math::log_modified_bessel_first_kind<std::vector<int, std::allocator<int> >, double&, (void*)0> (b=@0x7fffffffb658: 1.8872158590898946, a=...) at stan/lib/stan_math/stan/math/prim/fun/log_modified_bessel_first_kind.hpp:241
#17 gpbf4_model_namespace::diagSPD_periodic<double, double, int, (void*)0> (alpha=@0x7fffffffb908: 4.3214919917410874, rho=@0x7fffffffb8f0: 0.72792931790474436, M=@0x555555843954: 20, pstream__=pstream__@entry=0x7fffffffbda0) at gpbf4.hpp:296
Which bottoms out in this call:
Eigen::Map<Eigen::Matrix<T_return, -1, 1>>(result.data(), result.size())
= x_vec.unaryExpr(
[f_ = std::forward<F>(f), y](auto&& v) { return f_(v, y); });
@SteveBronder thoughts? Is this missing a forward on v?
Oh, I think it was a few lines above that snippet, we're reading x.size after we move x into x_vec
RC4 released Friday. We could release this Friday, but I generally prefer releasing at the starts of weeks, so I'd suggest the 2nd
It's been a week, and nothing seems to have caught fire, so I'm going to close this issue and say this release is done!
We're going to schedule a meeting (sometime in the next couple months) to discuss how this release went and talk about how we want to schedule and organize releases going forward. If people are interested in joining, let me know. The result will probably be something that ends up in stan-dev/design-docs to both write down our current process and discuss any tweaks we want to make at this point