Windows Error: install_cmdstan

[brianperdomo]

Describe the bug

I'm attempting a fresh installation of cmdstanr on Windows and I get the following error: (FYI: The overwrite is because I tried again after the first error)

cmdstanr::install_cmdstan(overwrite = TRUE)

The C++ toolchain required for CmdStan is setup properly!
trying URL 'https://api.github.com/repos/stan-dev/cmdstan/releases/latest'
downloaded 15 KB

* Latest CmdStan release is v2.35.0
* Installing CmdStan v2.35.0 in C:\Users\brian/.cmdstan/cmdstan-2.35.0
* Downloading cmdstan-2.35.0.tar.gz from GitHub...
* Removing the existing installation of CmdStan...
trying URL 'https://github.com/stan-dev/cmdstan/releases/download/v2.35.0/cmdstan-2.35.0.tar.gz'
Content type 'application/octet-stream' length 51682840 bytes (49.3 MB)
downloaded 49.3 MB

* Download complete
* Unpacking archive...
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file 'C:\Users\brian/.cmdstan/cmdstan-2.35.0/make/local': No such file or directory

I checked that path and the problem is that the actual path should be "C:\Users\brian\.cmdstsan\cmdstan-2.35.0\cmdstan-2.35.0\make\local"

I've tried to find ways to get it to point to the right directory, but I'm at a loss.

To Reproduce cmdstan::install_cmdstan()

Expected behavior Either the archive needs to be unpacked in a way that matches the expected cmdstan_path() without the duplication of cmdstan-2.35.0 or cmdstan_path() should point to the path with the duplicate.... probably the former makes more sense.

Operating system Windows 11

CmdStanR version number

tried with the version 0.8.1 from CRAN and the development version from github

Additional context R 4.4.1, Rtools44

[jgabry]

So is the difference in the paths just the slashes? I'm not a Windows user, so maybe @rok-cesnovar or @andrjohns will have an idea about why that's happening.

I checked that path and the problem is that the actual path should be "C:\Users\brian.cmdstsan\cmdstan-2.35.0\cmdstan-2.35.0\make\local"

I assume .cmdstsan is a typo or do you really have an extra "s" in the path? That would definitely cause a problem, but guessing that's just a typo.

[jgabry]

Oh actually I see there’s two instances of cmdstan-2.35.0 in the path that you say is the correct one. That’s strange. Do you know why that’s the case?

[brianperdomo]

Sorry, the .cmdstsan was a typo. The issue is the duplication of the cmdstan-2.35.0 directory in the path.

I have not been able to figure out the issue. I'll try to take a look this weekend to see if I can find other clues. Are there any settings or other diagnostics I should check that could be informative?

[jgabry]

I don’t think I’ve seen this particular issue with the path before. A couple things to try if you’re not able to figure out why that’s happening:

• Delete the entire .cmdstan folder. Then rerun install_cmdstan(). Maybe you tried this already, if not worth a try.

• download CmdStan yourself and unpack it yourself. Then point cmdstanr to the location using set_cmdstan_path() and then run rebuild_cmdstan().

[brianperdomo]

Deleting .cmdstan and repeating the install function did not work, but I was able to unpack and rebuild cmdstan like you suggested. Thanks!

[jgabry]

I'm still not sure what the source of the problem is, but glad you got it running successfully using that workaround!

[svethitov]

Hi @jgabry,

I also managed to make it work with the workaround.

It appears that the problem is with the utils::untar function call at R/install.R. Apparently, it is not correctly executing with the --strip-components option on Windows 11. Because of that you keep the parent folder from the archive and you get this nested cmdstan-2.35.0/cmdstan-2.35.0 structure.

You can reproduce it by simply trying to untar the release manually:

utils::untar(
  "cmdstan-2.35.0.tar.gz",
  exdir = "cmdstan-2.35.0",
  extras = "--strip-components 1"
)

And the reason why it is not correctly executing is that Sys.getenv("TAR") is equal to "internal" by default on my system. Apparently, on the Windows 11 systems as a whole. However, this leads utils::untar to default to calling untar2(tarfile, files, list, exdir, restore_times). As you can see from this call it is not taking the extra parameters (extras) into account. You can solve the issue by setting the tar executable (parameter tar of utils::untar) to tar.exe, which should be installed on Windows.

utils::untar(
  "cmdstan-2.35.0.tar.gz",
  exdir = "extract_dir_11",
  extras = "--strip-components 1",
  tar = "tar.exe"
)

This last call worked for me and it is correctly passing extra options to the unzipping, which is producing the correct folder structure.

So I suppose something like:

utils::untar(
  dest_file,
  exdir = dir_cmdstan,
  extras = "--strip-components 1",
  tar = if (os_is_windows()) "tar.exe" else Sys.getenv("TAR")
)

will do the trick.

However, another workaround is just to change the "TAR" environmental variable prior to running install_cmdstan(). I tried it and it work.

[jgabry]

@svethitov Thanks for these details, super helpful! I just created https://github.com/stan-dev/cmdstanr/pull/1031 using your suggestion.

[WardBrian]

Some notes:

Sys.getenv("TAR") is equal to "internal" by default on my system. Apparently, on the Windows 11 systems as a whole

I think this is almost certainly some R funny business, TAR is not one of the environment variables that the operating system itself sets by default. "internal" also happens to be the exact special string that untar expects when requesting to not use an external tar program

tar.exe, which should be installed on Windows.

This is only true for relatively recent installations (circa April 2018)

BSD TAR.exe was added to Windows 10 (1803) from build 17063 or later.

Because it's BSD, I think it also might not support the --strip-components flag. It does sound like Rtools includes a tar.exe, which would work as long as it appears higher in the PATH.

---

I guess one question @jgabry is why doesn't exdir just point to the parent directory (without the cmdstan_vXXX)? That would allow you to drop the --strip-components flag entirely, and then it wouldn't matter which tar implementation gets used, including internal. That way the testing/verifying the change is platform independent

[jgabry]

Thanks @WardBrian.

I guess one question @jgabry is why doesn't exdir just point to the parent directory (without the cmdstan_vXXX)? That would allow you to drop the --strip-components flag entirely, and then it wouldn't matter which tar implementation gets used, including internal. That way the testing/verifying the change is platform independent

That's a good question, I'm not sure why (I think @rok-cesnovar would know, perhaps there was a specific reason). In other words, you think we can just use

utils::untar(dest_file, exdir = dir)

where dest_file is the path to the tar.gz file and dir is either the default directory (".cmdstan" directory within the user's home directory) or a directory specified by the user. Is that right? If that works in general that sounds like a good idea.

[WardBrian]

Yes -- That is what cmdstanpy does, essentially. You might need to check if this folder already exists, I'm not sure what the behavior is in that case, but the code might already be checking if the requested version is installed somewhere?

[jgabry]

You might need to check if this folder already exists, I'm not sure what the behavior is in that case

Yeah, if the default directory is used we check that it exists and if not we create it. If they specify a custom directory we check that it exists:

https://github.com/stan-dev/cmdstanr/blob/b762c54a752fab0bedc7951913186861b4706079/R/install.R#L117-L125

but the code might already be checking if the requested version is installed somewhere?

If we find the requested version already installed in the specified location we just throw an error and tell them to delete it or specify overwrite=TRUE when calling install_cmdstan.

I went ahead and made a PR (https://github.com/stan-dev/cmdstanr/pull/1034) with your suggested change, so that should test this on all the GitHub actions configurations we're using, although I don't think that includes old versions of Windows. But, if I understand correctly, for this change it shouldn't matter which version of Windows the user has. Thanks again for your help with this.

[WardBrian]

The Github Actions environments are not always representative of a typical user, unfortunately (especially on Windows), but you're correct that one of the goals of my suggestion was to get out of R's way and let it handle the platform specifics. The real test would probably be a user like @svethitov giving that branch a try

[jgabry]

@svethitov and/or @brianperdomo would you by chance be able to test the fix I implemented using @WardBrian's suggestion? You can install cmdstanr from that branch using

remotes::install_github("stan-dev/cmdstanr", ref = "fix-untar-install")

and then just run install_cmdstan().

[brianperdomo]

Worked for me!

[svethitov]

@jgabry it works for me as well. The change suggested by @WardBrian makes much more sense than hacking a Windows only solution with the environmental variables.

[jgabry]

Great, thank you both for testing it!