loading python package via reticulate fails with `rmarkdown::render` whereas `Rscript` succeeds

[bobermayer]

Hi,

I'm trying to run python packages via reticulate, but imports fail when using rmarkdown::render("script.R") whereasRscript script.R succeeds.

here's my conda env

mamba create -c conda-forge -n test scanpy pandoc

here's my script test_env.R

library(methods)
library(futile.logger)
conda_home <- '/home/obermayb_c/miniconda3'
conda_env <- 'test'
knitr::opts_chunk$set(echo = TRUE, cache=TRUE, cache.lazy=FALSE, message=FALSE, warning=FALSE,
                      engine.path = list(python = file.path(conda_home,"envs",conda_env,"bin","python")))
Sys.setenv(RETICULATE_PYTHON=file.path(conda_home,'envs',conda_env,'bin','python'))
Sys.setenv(LD_LIBRARY_PATH=file.path(conda_home,'envs',conda_env,'lib'))
library(reticulate)
use_python(file.path(conda_home,"envs",conda_env,'bin','python'))
use_condaenv(file.path(conda_home,"envs",conda_env), conda = file.path(conda_home,"condabin/conda"))
flog.info(.libPaths())
flog.info(paste("LD_LIBRARY_PATH: ",Sys.getenv("LD_LIBRARY_PATH")))
flog.info(py_config())
scanpy <- reticulate::import('scanpy', convert=TRUE)

here's the output of Rscript test_env.R:

INFO [2024-02-26 13:36:13] /home/obermayb_c/R/x86_64-pc-linux-gnu-library/4.0
INFO [2024-02-26 13:36:13] /opt/R/4.0.3/lib/R/library
INFO [2024-02-26 13:36:13] LD_LIBRARY_PATH:  /home/obermayb_c/miniconda3/envs/test/lib
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/bin/python
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/lib/libpython3.12.so
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test:/home/obermayb_c/miniconda3/envs/test
INFO [2024-02-26 13:36:13] /home/obermayb_c/R/x86_64-pc-linux-gnu-library/4.0/reticulate/config:/home/obermayb_c/miniconda3/envs/test/lib/python312.zip:/home/obermayb_c/miniconda3/envs/test/lib/python3.12:/home/obermayb_c/miniconda3/envs/test/lib/python3.12/lib-dynload:/home/obermayb_c/miniconda3/envs/test/lib/python3.12/site-packages
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test
INFO [2024-02-26 13:36:13] 
INFO [2024-02-26 13:36:13] 
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/bin/python
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/bin/python
INFO [2024-02-26 13:36:13] 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:50:58) [GCC 12.3.0]
INFO [2024-02-26 13:36:13] 3.12
INFO [2024-02-26 13:36:13] 64bit
INFO [2024-02-26 13:36:13] FALSE
INFO [2024-02-26 13:36:13] True
INFO [2024-02-26 13:36:13] list(path = "/home/obermayb_c/miniconda3/envs/test/lib/python3.12/site-packages/numpy", version = list(c(1, 26, 4)))
INFO [2024-02-26 13:36:13] NULL
INFO [2024-02-26 13:36:13] NULL
INFO [2024-02-26 13:36:13] TRUE
INFO [2024-02-26 13:36:13] /home/obermayb_c/miniconda3/envs/test/bin/python
INFO [2024-02-26 13:36:13] RETICULATE_PYTHON

and here's the result of Rscript -e "library(rmarkdown); rmarkdown::render('test_env.R','html_document')" :

processing file: test_env.spin.Rmd
  |...................................                 |  67% [unnamed-chunk-1]
Quitting from lines 3-19 [unnamed-chunk-1] (test_env.spin.Rmd)
Error in `py_module_import()`:
! ImportError: /home/obermayb_c/miniconda3/envs/test/lib/python3.12/site-packages/PIL/../../../libtiff.so.6: undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0
Run `reticulate::py_last_error()` for details.
Backtrace:
 1. reticulate::import("scanpy", convert = TRUE)
 2. reticulate:::py_module_import(module, convert = convert)
                                                                                                            
Execution halted

uncommenting the reticulate::import line gives the exact same logs, so library paths etc. should all be identical as far as I can see

it's somehow related to libjpeg-turbo, but I can't figure out why there's a difference between the Rscript and the rmarkdown::render calls.

sessionInfo

> xfun::session_info("rmarkdown")
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.4 LTS, RStudio 2023.12.1.402

Locale:
  LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
  LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

Package version:
  base64enc_0.1.3   bslib_0.6.1       cachem_1.0.8      cli_3.6.2         digest_0.6.34     ellipsis_0.3.2   
  evaluate_0.23     fastmap_1.1.1     fontawesome_0.5.2 fs_1.6.3          glue_1.7.0        graphics_4.0.3   
  grDevices_4.0.3   highr_0.10        htmltools_0.5.7   jquerylib_0.1.4   jsonlite_1.8.8    knitr_1.45       
  lifecycle_1.0.4   magrittr_2.0.3    memoise_2.0.1     methods_4.0.3     mime_0.12         R6_2.5.1         
  rappdirs_0.3.3    rlang_1.1.3       rmarkdown_2.25    sass_0.4.8        stats_4.0.3       stringi_1.8.3    
  stringr_1.5.1     tinytex_0.49      tools_4.0.3       utils_4.0.3       vctrs_0.6.5       xfun_0.42        
  yaml_2.3.8       

Pandoc version: 3.1.12.1

Checklist

When filing a bug report, please check the boxes below to confirm that you have provided us with the information we need. Have you:

• [x] formatted your issue so it is easier for us to read?

• [x] included a minimal, self-contained, and reproducible example?

• [x] pasted the output from xfun::session_info('rmarkdown') in your issue?

• [x] upgraded all your packages to their latest versions (including your versions of R, the RStudio IDE, and relevant R packages)?

• [x] installed and tested your bug with the development version of the rmarkdown package using remotes::install_github("rstudio/rmarkdown")?