rdotnet In R-4.2.0 on windows, package 'stats' in options("defaultPackages") was not found.

Using R version 4.2.0 on Windows, this prior issue with loading stats.dll https://github.com/rdotnet/rdotnet/issues/127 seems to have resurfaced.

Here, I show that rnorm works with R 4.0.2 but fails with R 4.2.0.

Works with old R version 4.0.2

#r "nuget: R.NET"
open RDotNet

REngine.SetEnvironmentVariables("c:/program files/r/r-4.0.2/bin/x64", "c:/program files/r/r-4.0.2")
let  engine = REngine.GetInstance()
engine.Evaluate("set.seed(1);rnorm(1)").AsNumeric()
// val it: NumericVector = seq [-0.6264538107]

Fails with R version 4.2.0

#r "nuget: R.NET"
open RDotNet

REngine.SetEnvironmentVariables("c:/program files/r/r-4.2.0/bin/x64", "c:/program files/r/r-4.2.0")
let  engine = REngine.GetInstance()
(*
During startup - Warning message:
package 'stats' in options("defaultPackages") was not found 
val engine: REngine
*)
engine.Evaluate("set.seed(1);rnorm(1)").AsNumeric()
(*
Error in rnorm(1) : could not find function "rnorm"
In addition: Warning message:
'memory.limit()' is no longer supported
RDotNet.EvaluationException: Error in rnorm(1) : could not find function "rnorm"

   at RDotNet.REngine.Parse(String statement, StringBuilder incompleteStatement, REnvironment environment)
   at RDotNet.REngine.Defer(String statement, REnvironment environment)+MoveNext()
   at System.Linq.Enumerable.TryGetLast[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.LastOrDefault[TSource](IEnumerable`1 source)
   at RDotNet.REngine.Evaluate(String statement, REnvironment environment)
   at <StartupCode$FSI_0005>.$FSI_0005.main@()
Stopped due to error
*)

> engine.Evaluate("library(stats)").AsCharacter()
(*
Error: package or namespace load failed for 'stats' in inDL(x, as.logical(local), as.logical(now), ...):
 unable to load shared object 'C:/Program Files/R/R-4.2.0/library/stats/libs/x64/stats.dll':
  LoadLibrary failure:  The specified module could not be found.
RDotNet.EvaluationException: Error: package or namespace load failed for 'stats' in inDL(x, as.logical(local), as.logical(now), ...):
 unable to load shared object 'C:/Program Files/R/R-4.2.0/library/stats/libs/x64/stats.dll':
  LoadLibrary failure:  The specified module could not be found.

   at RDotNet.REngine.Parse(String statement, StringBuilder incompleteStatement, REnvironment environment)
   at RDotNet.REngine.Defer(String statement, REnvironment environment)+MoveNext()
   at System.Linq.Enumerable.TryGetLast[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.LastOrDefault[TSource](IEnumerable`1 source)
   at RDotNet.REngine.Evaluate(String statement, REnvironment environment)
   at <StartupCode$FSI_0006>.$FSI_0006.main@()
Stopped due to error
*)

Apr 25 '22 11:04 nhirschey

Thanks for reporting @nhirschey! I'm seeing the same thing and will investigate a solution.

Apr 29 '22 20:04 lrasmus

Just some placeholder notes/updates -

Going off some other/older posts, it seems like stats.dll is failing to load because of some dependency not being found. When you load it in dependency walker, you can see R.dll, Rblas.dll, and Rlapack.dll are dependencies, all located (for me) in C:\Program Files\R\R-4.2.0\bin\x64. I've been looking at the way (per #127) that Renviron is loaded, how PATH is being set, etc, because the behavior is like the dependency can't be found on the path when trying to load. Playing around with a suggestion in https://stackoverflow.com/a/48211343/5670646 (NOTE I don't recommend this as a solution, I was just doing these steps to troubleshoot), once Rblas.dll and Rlapack.dll are put in the same spot as stats.dll, that error goes away.

I need to go back to why, even if I explicitly set the folder where those dependency DLLs are found, it's failing to load. There's probably something in the basics of how DLLs are loaded that I'm just forgetting.

May 03 '22 20:05 lrasmus

More placeholder notes (sorry) -

Thanks to @nmunozgarcia for noticing the connection in #152. It seems similar - setting PATH explicitly doesn't seem to fix the issue, which is odd because it should. But that's similar to the behavior folks are seeing with R_HOME - they set it in the environment, but R embedded doesn't pick it up. So current theory (which is a good one) would be that PATH updates are similarly not being picked up/recognized. This assumes that the root cause is the PATH needs to be set so it can "find" the core R dlls.

I did test explicitly setting my PATH in .NET code, and I confirmed within R.NET right before it calls R that the environment variable is there as I expected - and it was. So I know PATH wasn't being overridden somewhere after I thought I set it.

Some references:

https://tdhock.github.io/blog/2020/embedded-R/ - Also notes about R_HOME issues
https://github.com/alexgutteridge/rsruby - Another embedded R
https://svn.r-project.org/R/trunk/src/unix/system.c - where the "R home" error occurs

May 11 '22 19:05 lrasmus

Not sure this is adding a lot of knowledge, but just confirmation via MSYS2 environment (from Rtools42) that unless the path is set to include the R core DLLs, it'll fail to load stats.dll

$ cd /C/Program\ Files/R/R-4.2.0/library/stats/libs/x64
$ ldd stats.dll
        ntdll.dll => /c/windows/SYSTEM32/ntdll.dll (0x7ff9c4870000)
        KERNEL32.DLL => /c/windows/System32/KERNEL32.DLL (0x7ff9c29d0000)
        KERNELBASE.dll => /c/windows/System32/KERNELBASE.dll (0x7ff9c24c0000)
        msvcrt.dll => /c/windows/System32/msvcrt.dll (0x7ff9c3550000)
        umppc15005.dll => /c/windows/System32/umppc15005.dll (0x22a939c0000)
        umppc15005.dll => /c/windows/System32/umppc15005.dll (0x22a939c0000)
        stats.dll => /c/Program Files/R/R-4.2.0/library/stats/libs/x64/stats.dll (0x7ff99df90000)
        ucrtbase.dll => /c/Windows/System32/ucrtbase.dll (0x7ff9c23c0000)
        R.dll => not found
        Rblas.dll => not found
        Rlapack.dll => not found
        api-ms-win-crt-private-l1-1-0.dll => not found
        api-ms-win-crt-string-l1-1-0.dll => not found
        api-ms-win-crt-environment-l1-1-0.dll => not found
        api-ms-win-crt-time-l1-1-0.dll => not found
        api-ms-win-crt-math-l1-1-0.dll => not found
        api-ms-win-crt-heap-l1-1-0.dll => not found
        api-ms-win-crt-runtime-l1-1-0.dll => not found
        api-ms-win-crt-stdio-l1-1-0.dll => not found


$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/progra~1/git/cmd:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl


$ export PATH=$PATH:/C/Program\ Files/R/R-4.2.0/bin/x64/

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/progra~1/git/cmd:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/C/Program Files/R/R-4.2.0/bin/x64/


$ ldd stats.dll
        ntdll.dll => /c/windows/SYSTEM32/ntdll.dll (0x7ff9c4870000)
        KERNEL32.DLL => /c/windows/System32/KERNEL32.DLL (0x7ff9c29d0000)
        KERNELBASE.dll => /c/windows/System32/KERNELBASE.dll (0x7ff9c24c0000)
        msvcrt.dll => /c/windows/System32/msvcrt.dll (0x7ff9c3550000)
        umppc15005.dll => /c/windows/System32/umppc15005.dll (0x1de77760000)
        umppc15005.dll => /c/windows/System32/umppc15005.dll (0x1de77760000)
        ucrtbase.dll => /c/windows/System32/ucrtbase.dll (0x7ff9c23c0000)
        R.dll => /c/Program Files/R/R-4.2.0/bin/x64/R.dll (0x7ff959eb0000)
        ADVAPI32.dll => /c/windows/System32/ADVAPI32.dll (0x7ff9c3ce0000)
        sechost.dll => /c/windows/System32/sechost.dll (0x7ff9c2930000)
        RPCRT4.dll => /c/windows/System32/RPCRT4.dll (0x7ff9c4700000)
        SHELL32.dll => /c/windows/System32/SHELL32.dll (0x7ff9c3f00000)
        msvcp_win.dll => /c/windows/System32/msvcp_win.dll (0x7ff9c2320000)
        USER32.dll => /c/windows/System32/USER32.dll (0x7ff9c2ff0000)
        win32u.dll => /c/windows/System32/win32u.dll (0x7ff9c1f90000)
        GDI32.dll => /c/windows/System32/GDI32.dll (0x7ff9c3e40000)
        gdi32full.dll => /c/windows/System32/gdi32full.dll (0x7ff9c20b0000)
        Rblas.dll => /c/Program Files/R/R-4.2.0/bin/x64/Rblas.dll (0x7ff9b2260000)
        Rlapack.dll => /c/Program Files/R/R-4.2.0/bin/x64/Rlapack.dll (0x7ff964370000)
        Riconv.dll => /c/Program Files/R/R-4.2.0/bin/x64/Riconv.dll (0x7ff9bde60000)
        Rgraphapp.dll => /c/Program Files/R/R-4.2.0/bin/x64/Rgraphapp.dll (0x7ff9abba0000)
        comdlg32.dll => /c/windows/System32/comdlg32.dll (0x7ff9c3470000)
        combase.dll => /c/windows/System32/combase.dll (0x7ff9c2a90000)
        shcore.dll => /c/windows/System32/shcore.dll (0x7ff9c3d90000)
        SHLWAPI.dll => /c/windows/System32/SHLWAPI.dll (0x7ff9c3720000)
        IMM32.dll => /c/windows/System32/IMM32.dll (0x7ff9c3cb0000)
        VERSION.dll => /c/windows/SYSTEM32/VERSION.dll (0x7ff9b7e40000)
        WINMM.dll => /c/windows/SYSTEM32/WINMM.dll (0x7ff9b73b0000)
        COMCTL32.dll => /c/windows/WinSxS/amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.19041.1110_none_792d1c772443f647/COMCTL32.dll (0x7ff9a1800000)
        MSIMG32.dll => /c/windows/SYSTEM32/MSIMG32.dll (0x7ff9b84e0000)

May 11 '22 20:05 lrasmus

Further confirmation R's view of PATH differs from the Environment's view of PATH

Environment.SetEnvironmentVariable("PATH", "C:/Program Files/R/R-4.2.0/bin/x64;" + Environment.GetEnvironmentVariable("PATH"));
REngine.SetEnvironmentVariables("c:/program files/r/r-4.2.0/bin/x64", "c:/program files/r/r-4.2.0");
Console.WriteLine(Environment.GetEnvironmentVariable("PATH"));

Produces:

c:/program files/r/r-4.2.0/bin/x64;C:/Program Files/R/R-4.2.0/bin/x64;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\R\R-4.2.0\bin\x64;C:\Program Files\SASHome\SASFoundation\9.4\core\sasexe;C:\Program Files\SASHome\SASFoundation\9.4\ets\sasexe;C:\Program Files\SASHome\Secure\ccme4;C:\Program Files\SASHome\x86\Secure\ccme4;C:\Program Files\SASHome\SASFoundation\9.4\core\sasext;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files\Azure Data Studio\bin;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\MATLAB\R2021b\bin;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Users\test\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\test\AppData\Local\Programs\Python\Python310\;C:\Users\test\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Azure Data Studio\bin;C:\Users\test\.dotnet\tools;

Note that my R directory exists in the PATH. Then we go ahead and load R.

var engine = REngine.GetInstance();

Calling Console.WriteLine(engine.Evaluate("Sys.getenv('PATH')").AsCharacter()[0]); does not have my R directory in the PATH anymore:

C:\rtools42/x86_64-w64-mingw32.static.posix/bin;C:\rtools42/usr/bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\R\R-4.2.0\bin\x64;C:\Program Files\SASHome\SASFoundation\9.4\core\sasexe;C:\Program Files\SASHome\SASFoundation\9.4\ets\sasexe;C:\Program Files\SASHome\Secure\ccme4;C:\Program Files\SASHome\x86\Secure\ccme4;C:\Program Files\SASHome\SASFoundation\9.4\core\sasext;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files\Azure Data Studio\bin;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\MATLAB\R2021b\bin;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Users\test\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\test\AppData\Local\Programs\Python\Python310\;C:\Users\test\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Azure Data Studio\bin;C:\Users\test\.dotnet\tools;

Finally, calling Console.WriteLine(Environment.GetEnvironmentVariable("PATH")); still has my R directory in the PATH:

c:/program files/r/r-4.2.0/bin/x64;C:\rtools42/x86_64-w64-mingw32.static.posix/bin;C:\rtools42/usr/bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\R\R-4.2.0\bin\x64;C:\Program Files\SASHome\SASFoundation\9.4\core\sasexe;C:\Program Files\SASHome\SASFoundation\9.4\ets\sasexe;C:\Program Files\SASHome\Secure\ccme4;C:\Program Files\SASHome\x86\Secure\ccme4;C:\Program Files\SASHome\SASFoundation\9.4\core\sasext;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files\Azure Data Studio\bin;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\MATLAB\R2021b\bin;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Users\test\AppData\Local\Programs\Python\Python310\Scripts\;C:\Users\test\AppData\Local\Programs\Python\Python310\;C:\Users\test\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Azure Data Studio\bin;C:\Users\test\.dotnet\tools;

May 11 '22 21:05 lrasmus

This workaround appears to fix the issue. I'm not satisfied yet though, because I want to know why this is happening!!

static void Main(string[] args)
{
    REngine.SetEnvironmentVariables("c:/program files/r/r-4.2.0/bin/x64", "c:/program files/r/r-4.2.0");
    var engine = REngine.GetInstance();

    // Workaround - explicitly include R libs in PATH so R environment can find them.  Not sure why R can't find them when
    // we set this via Environment.SetEnvironmentVariable
    engine.Evaluate("Sys.setenv(PATH = paste(\"C:/Program Files/R/R-4.2.0/bin/x64\", Sys.getenv(\"PATH\"), sep=\";\"))");

    // This now works
    engine.Evaluate("library(stats)");
    engine.Evaluate("set.seed(1);rnorm(1)").AsNumeric();
}

May 11 '22 21:05 lrasmus

Thanks for the workaround, that's really nice! Regarding where it's coming from, here's a long shot: this stack overflow comment talks about some ways that R PATH being set on startup.

I searched the referenced folder, C:\Program Files\R\R-4.2.0\etc, for PATH and I can see the path being set in the Rcmd_environ file. Is rdotnet calling Rcmd?

C:\Program Files\R\R-4.2.0\etc> ls * | select-string "PATH"

Rcmd_environ:38:R_RTOOLS42_PATH="${RTOOLS42_HOME:-c:/rtools42}/x86_64-w64-mingw32.static.posix/bin;${RTOOLS42_HOME:-c:/rtools42}/usr/bin"
Rcmd_environ:39:#   trailing slash to protect against trailing backslash in PATH
Rcmd_environ:40:PATH="${R_CUSTOM_TOOLS_PATH:-${R_RTOOLS42_PATH}};${PATH}/"
Rprofile.site:12:# .Library.site <- file.path(chartr("\\", "/", R.home()), "site-library")

May 12 '22 00:05 nhirschey

Awesome find @nhirschey!!! Thanks so much. rdotnet doesn't itself explicitly call Rcmd_environ. I tried editing that file to update the PATH, and it didn't appear to pick up my changed path. Even going so far as to remove that file entirely didn't seem to make a difference for when rdotnet runs. I think that is only used for R CMD invocation.

But along those lines, I think the PATH changes are being made in C:\Program Files\R\R-4.2.0\library\base\R\Rprofile. At the bottom of that file, there is code to update the path with RTools. I can also edit it as such and have it work:

local({
    setRtools42Path <- 0
     setRtools42Path <- 1

    if (setRtools42Path) {
        rthome <- Sys.getenv("RTOOLS42_HOME", "c:/rtools42")
        rtpath <- paste0(rthome, "/x86_64-w64-mingw32.static.posix/bin;",
                         rthome, "/usr/bin")
        path <- Sys.getenv("R_CUSTOM_TOOLS_PATH", rtpath)
	## ADDED rbin
	rbin <- paste0(Sys.getenv("R_HOME"), "/bin/x64")
        Sys.setenv(R_RTOOLS42_PATH = rtpath)
	## UPDATED
        Sys.setenv(PATH = paste0(rbin, ";", path, ";", Sys.getenv("PATH")))
    }
})

May 13 '22 15:05 lrasmus

I'm still confused why setting the PATH environment variable in the C# code had no effect. I know that loaded processes inherit a copy of their parent's environment, so if R.dll was loaded and THEN we update PATH, it makes sense that R.dll wouldn't know about those changes. But, it looks like R.dll is loaded AFTER I was updating PATH. I verified this via ListDLLs by setting a breakpoint in my C# test app before any rdotnet methods are called. I run listdlls TestApp.exe and confirm that R.dll isn't loaded. Then I continue on to initialize the R engine, and verify using listdlls TestApp.exe that R.dll is now loaded.

May 13 '22 15:05 lrasmus

Workarounds that are available:

Add C:\Program Files\R\R-4.2.0\bin\x64 to the PATH environment variable using the standard Windows System Properties > Environment Variables
Update C:\Program Files\R\R-4.2.0\library\base\R\Rprofile (per earlier comment)
In calling application, immediately call engine.Evaluate("Sys.setenv(PATH = paste(\"C:/Program Files/R/R-4.2.0/bin/x64\", Sys.getenv(\"PATH\"), sep=\";\"))"); after initializing REngine (per earlier comment)

May 13 '22 15:05 lrasmus

Just to add some info, I am using R 4.2.2 and workaround 3 didnt work, but workaround 2 worked like a charm.

Feb 24 '23 00:02 gkapellmann

Just to add some info, I am using R 4.2.2 and workaround 3 didnt work, but workaround 2 worked like a charm.

Please, how did you setup your F# project and get it to work. I am having no luck.

Mar 07 '23 05:03 robreno

rdotnet rdotnet copied to clipboard

In R-4.2.0 on windows, package 'stats' in options("defaultPackages") was not found.

Works with old R version 4.0.2

Fails with R version 4.2.0

rdotnet
rdotnet copied to clipboard