stan-dev
include_paths dropped for pre-compiled models in subdirectories
Apologies if this is a duplicate of #550 or #687, but I have a case where include_paths is not recognized for a pre-compiled model in a subdirectory. Users of instantiate are having trouble building packages with pre-compiled models split over multiple Stan files (c.f. https://github.com/wlandau/instantiate/discussions/28).
Here's the full context. With these files:
fs::dir_tree("bin")
bin
└── stan
├── bernoulli.stan
└── utils
└── silly.stan
bin/stan/bernoulli.stan:
#include utils/silly.stan
data {
int<lower=0> N;
array[N] int<lower=0,upper=1> y;
}
parameters {
real<lower=0,upper=1> theta;
}
model {
theta ~ beta(1,1); // uniform prior on interval 0,1
y ~ bernoulli(theta);
}
generated quantities {
real theta_lin;
theta_lin = silly_logit(theta);
}
and bin/stan/utils/silly.stan:
functions {
real silly_logit(real x) {
real out;
out = logit(x);
return out;
}
}
and R code to compile and reuse the model:
library(cmdstanr)
#> This is cmdstanr version 0.9.0
#> - CmdStanR documentation and vignettes: mc-stan.org/cmdstanr
#> - CmdStan path: /Users/C240390/.cmdstan/cmdstan-2.36.0
#> - CmdStan version: 2.36.0
model <- cmdstan_model(
stan_file = "bin/stan/bernoulli.stan",
exe_file = "bin/stan/bernoulli",
compile = TRUE,
include_paths = "bin/stan"
)
rstudioapi::restartSession()
library(cmdstanr)
model <- cmdstan_model(
stan_file = "bin/stan/bernoulli.stan",
exe_file = "bin/stan/bernoulli",
compile = FALSE,
include_paths = "bin/stan"
)
y <- c(0, 1, 1, 0, 1, 0)
samples <- model$sample(data = list(N = length(y), y = y))
I get this error:
Error in `(function (command = NULL, args = character(), error_on_status = TRUE, …`:
! System command 'stanc' failed
---
Exit status: 1
Stderr (last 10 lines, see `$stderr` for more):
34: */
35:
36: #include utils/silly.stan
^
37: data {
38: int<lower=0> N;
-------------------------------------------------
Could not find include file 'utils/silly.stan' in specified include paths.
Current include paths: None
Same if I set include_paths to "bin/stan/utils", file.path(getwd(), "bin/stan"), or file.path(getwd(), "bin/stan/utils").
R version 4.5.0 (2025-04-11)
Platform: aarch64-apple-darwin20
Running under: macOS Sequoia 15.5
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.1
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
time zone: America/Indiana/Indianapolis
tzcode source: internal
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] cmdstanr_0.9.0
loaded via a namespace (and not attached):
[1] tensorA_0.36.2.1 backports_1.5.0 R6_2.6.1 xfun_0.52
[5] magrittr_2.0.3 posterior_1.6.1 glue_1.8.0 abind_1.4-8
[9] tibble_3.2.1 knitr_1.50 pkgconfig_2.0.3 generics_0.1.3
[13] lifecycle_1.0.4 ps_1.9.1 cli_3.6.4 processx_3.8.6
[17] vctrs_0.6.5 withr_3.0.2 compiler_4.5.0 distributional_0.5.0
[21] rstudioapi_0.17.1 tools_4.5.0 checkmate_2.3.2 evaluate_1.0.3
[25] pillar_1.10.2 jsonlite_2.0.0 rlang_1.1.6
Thanks for reporting this. Can you try removing the stan_file argument (which isn't strictly necessary) when you provide exe_file? So, the second call to cmdstan_model() would be:
model <- cmdstan_model(
exe_file = "bin/stan/bernoulli",
compile = FALSE,
include_paths = "bin/stan"
)
y <- c(0, 1, 1, 0, 1, 0)
samples <- model$sample(data = list(N = length(y), y = y))
When I do that it samples fine. I'm not yet sure why that's the case (maybe @andrjohns or @SteveBronder has an idea), but curious if that also works for you.
We'll need to look into why it errors when stan_file is also specified, but I'm glad that there's at least a way to get this to work in the meantime.
I may have a fix for this. Does installing cmdstanr from the branch exe-w-include_paths solve the problem (ideally without creating new ones!)? It seems to work when I test it on your example, but it would be great if you (and/or one of the instantiate users who encountered this problem) could try it.
and/or one of the instantiate users who encountered this problem
What about you, @lcgodoy? @Abrahamiaan?
and/or one of the instantiate users who encountered this problem
What about you, @lcgodoy? @Abrahamiaan?
It did not work for me. First, I tried this example again. It installs the package without any problems, but the tests fail.
Next, I tried following the same steps you did here, that is:
library(cmdstanr)
list.files(system.file("bin/stan", package = "example"))
my_path <- system.file("bin/stan", package = "example")
model <- cmdstan_model(
stan_file = file.path(my_path, "bernoulli.stan"),
exe_file = file.path(my_path, "bernoulli"),
compile = FALSE,
include_paths = file.path(my_path, "utils")
)
y <- c(0, 1, 1, 0, 1, 0)
samples <- model$sample(data = list(N = length(y), y = y))
and I got:
> samples <- model$sample(data = list(N = length(y), y = y))
Error in `(function (command = NULL, args = character(), error_on_status = TRUE, …`:
! System command 'stanc' failed
---
Exit status: 1
Stderr (last 10 lines, see `$stderr` for more):
34: */
35: functions {
36: #include utils/silly.stan
^
37: }
38: data {
-------------------------------------------------
Could not find include file 'utils/silly.stan' in specified include paths.
Current include paths: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library/example/bin/stan/utils
---
Type .Last.error to see the more details.
Here goes my session info:
R> sessionInfo()
R version 4.4.1 (2024-06-14)
Platform: aarch64-apple-darwin20
Running under: macOS Sonoma 14.6.1
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.0
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
time zone: America/Los_Angeles
tzcode source: internal
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] cmdstanr_0.9.0.9000
loaded via a namespace (and not attached):
[1] tensorA_0.36.2.1 backports_1.5.0 R6_2.6.1 xfun_0.45
[5] magrittr_2.0.3 posterior_1.6.1 glue_1.8.0 abind_1.4-8
[9] tibble_3.2.1 knitr_1.47 pkgconfig_2.0.3 generics_0.1.4
[13] lifecycle_1.0.4 ps_1.9.1 cli_3.6.5 processx_3.8.6
[17] vctrs_0.6.5 withr_3.0.2 compiler_4.4.1 tools_4.4.1
[21] distributional_0.5.0 checkmate_2.3.2 pillar_1.10.2 jsonlite_2.0.0
[25] rlang_1.1.6
I just realized that by removing the stan_file argument:
model <- cmdstan_model(
exe_file = file.path(my_path, "bernoulli"),
compile = FALSE,
include_paths = file.path(my_path, "utils")
)
Sampling works smoothly.
Thanks.
@lcgodoy, did you try @jgabry's patch?
pak::pkg_install("stan-dev/cmdstanr@exe-w-include_paths")
I would prefer to keep the stan_file argument in instantiate to preserve the option to re-compile the model in the package.
@lcgodoy, did you try @jgabry's patch?
pak::pkg_install("stan-dev/cmdstanr@exe-w-include_paths") I would prefer to keep the
stan_fileargument ininstantiateto preserve the option to re-compile the model in the package.
Yes, I did. It compiles and works as in here. But when I try to use the pre-compiled models via instantiate::stan_package_model, it fails.