azureml-sdk-for-r icon indicating copy to clipboard operation
azureml-sdk-for-r copied to clipboard

Published 20 hours ago • verified publisher icon Azure

adding CRAN packages to compute cluster experiment causes error

Open krenova opened this issue 4 years ago • 8 comments

Main Issue Have been trying do use the "xgboost" package to do predictive modelling on the compute-cluster but find myself facing the following fatal error messages

---------------------------------------------------------
----- Preparing Workspace
---------------------------------------------------------
AttributeError: module 'azureml' has no attribute 'core'Error in value[[3L]](cond) : 
Calls: get_workspace ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted
2020/08/21 08:08:31 logger.go:293: Failed to run the wrapper cmd with err: exit status 1
2020/08/21 08:08:31 sysutils_linux.go:221: mpirun version string: {
mpirun (Open MPI) 3.1.2

Report bugs to http://www.open-mpi.org/community/help/
}
2020/08/21 08:08:31 logger.go:293: Process Exiting with Code:  1

In AML platform, I also encountered a notification indicating failure is due to image

In the azureml-logs, i noticed

ActivityFailedException: ActivityFailedException:
	Message: Activity Failed:
{
    "error": {
        "code": "ServiceError",
        "message": "Job release failed: No module named 'ruamel'",
        "details": [],
        "debugInfo": {
            "type": "ModuleNotFoundError",
            "message": "No module named 'ruamel'"

Main Code The codes I used are almost exactly similar to those given by the iris dataset example: https://github.com/Azure/azureml-sdk-for-r/tree/master/samples/training/train-on-amlcompute

The only thing i did different was to add a list to the cran_packages option.

est <- estimator(
  source_directory = ".",
  entry_script = "train.R",
  cran_packages = list("xgboost"),
  compute_target = compute_target
)

I understand from the [https://azure.github.io/azureml-sdk-for-r/reference/cran_package.html](R package website) that we're supposed to use the function cran_package() to define the require packages instead but it appears that this function does not exist. In fact, it appears that the code defining the estimator function in my R and the one on github is very different.

My Session Info

R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18363)

Matrix products: default

locale:
[1] LC_COLLATE=English_Singapore.1252  LC_CTYPE=English_Singapore.1252    LC_MONETARY=English_Singapore.1252
[4] LC_NUMERIC=C                       LC_TIME=English_Singapore.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] jsonlite_1.6.1    azuremlsdk_0.6.85

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.5          rstudioapi_0.11     knitr_1.28          servr_0.16          magrittr_1.5        rappdirs_0.3.1     
 [7] xtable_1.8-4        lattice_0.20-38     R6_2.4.1            rlang_0.4.6         fastmap_1.0.1       tools_3.6.1        
[13] DT_0.13             grid_3.6.1          xfun_0.13           htmltools_0.4.0     crosstalk_1.1.0.1   digest_0.6.25      
[19] Matrix_1.2-17       shiny_1.4.0.2       later_1.0.0         htmlwidgets_1.5.1   promises_1.1.0      evaluate_0.14      
[25] mime_0.9            rmarkdown_2.1       compiler_3.6.1      shinycssloaders_0.3 reticulate_1.15     httpuv_1.5.2

krenova avatar Aug 21 '20 17:08 krenova

Hi @krenova this is recent issue we've just seen due to an upgrade to pip. A workaround is to pin pip to version 20.1.1. You should be able to do so by updating your env like so:

env <- r_environment(name = "my-env")
env$python$conda_dependencies$add_conda_package('pip==20.1.1')

Or you could create your own image, see the vignette here (this is a bit more involved):

  • https://github.com/Azure/azureml-sdk-for-r/pull/379

mx-iao avatar Aug 24 '20 16:08 mx-iao

@mx-iao I tried env$python$conda_dependencies$add_conda_package('pip==20.1.1') but unfortunately the same errors surfaced.

I tried uninstalling and installing/updating the ruamel package but it didnt change anything.

r_env$python$conda_dependencies$add_conda_package('pip==20.1.1')
r_env$python$conda_dependencies$remove_pip_package('ruamel.yaml')
r_env$python$conda_dependencies$add_pip_package('ruamel.yaml')

Could I have missed something?

Thanks so much for pointing to the vignette. Will be looking to set up a docker image with the compute instance but expecting the approvals process in the company i'm consulting for to take some time. Is there an alternate workaround for now?

krenova avatar Aug 26 '20 08:08 krenova

@krenova Are you using the SDK installed from GitHub? If not can you first upgrade to that: https://azure.github.io/azureml-sdk-for-r/articles/installation.html#install-the-azuremlsdk-r-package

mx-iao avatar Aug 26 '20 18:08 mx-iao

Hi mx-iao,

@krenova and I working as a team. I followed your instructions.

Encountered an error when R package data.table is installed and compiled from source.

The command '/bin/sh -c R -e "library(checkpoint); snapshot_date <- tail(checkpoint::getValidSnapshots(), n = 1); setSnapshot(snapshot_date); install.packages(c('reticulate', 'remotes', 'e1071', 'optparse')); library(remotes); remotes::install_cran('azuremlsdk', upgrade = FALSE); remotes::install_version('data.table', version = '1.12.8'); remotes::install_version('xgboost', version = '0.82.1'); remotes::install_version('e1071'); remotes::install_github('Azure/azureml-sdk-for-r')"' returned a non-zero code: 1 2020/09/18 05:02:38 Container failed during run: acb_step_0. No retries remaining.

How to proceed?

Herman

The docker build logs: [0m[91m* installing source package ‘data.table’ ... [0m[91m** package ‘data.table’ successfully unpacked and MD5 sums checked [0m[91m** using staged installation [0m*** pkg-config is not installed. *** Compilation will now be attempted and if it works you can ignore this message. However, *** if compilation fails, try 'locate zlib.h zconf.h' and ensure the zlib development library *** is installed : *** deb: zlib1g-dev (Debian, Ubuntu, ...) *** rpm: zlib-devel (Fedora, EPEL, ...) *** brew: zlib (OSX) *** Note that zlib is required to compile R itself so you may find the advice in the R-admin *** guide helpful regarding zlib. On Debian/Ubuntu, zlib1g-dev is a dependency of r-base as *** shown by 'apt-cache showsrc r-base | grep ^Build-Depends | grep zlib', and therefore *** 'sudo apt-get build-dep r-base' should be sufficient too. *** To silence this message, please ensure that : *** 1) 'pkg-config --exists zlib' succeeds (i.e. $? -eq 0) *** 2) 'pkg-config --libs zlib' contains -lz *** Compilation will now be attempted ... [91m** libs [0mx86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c assign.c -o assign.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c between.c -o between.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c bmerge.c -o bmerge.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c chmatch.c -o chmatch.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c cj.c -o cj.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c coalesce.c -o coalesce.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c dogroups.c -o dogroups.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c fastmean.c -o fastmean.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c fcast.c -o fcast.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c fifelse.c -o fifelse.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c fmelt.c -o fmelt.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c forder.c -o forder.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c frank.c -o frank.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c fread.c -o fread.o [91mfread.c: In function 'freadMain': [0m[91mfread.c:1293:5: warning: ignoring return value of 'strtod', declared with attribute warn_unused_result [-Wunused-result] (void)strtod(ch, &end); // careful not to let "" get to here (see continue above) as strtod considers "" numeric ^~~~~~~~~~~~~~~~~~~~~~ [0mx86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c freadR.c -o freadR.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c froll.c -o froll.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c frollR.c -o frollR.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c frolladaptive.c -o frolladaptive.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c fsort.c -o fsort.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c fwrite.c -o fwrite.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c fwriteR.c -o fwriteR.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c gsumm.c -o gsumm.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c ijoin.c -o ijoin.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c init.c -o init.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c inrange.c -o inrange.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c nafill.c -o nafill.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c nqrecreateindices.c -o nqrecreateindices.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c openmp-utils.c -o openmp-utils.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c quickselect.c -o quickselect.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c rbindlist.c -o rbindlist.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c reorder.c -o reorder.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c shift.c -o shift.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c snprintf.c -o snprintf.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c subset.c -o subset.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c transpose.c -o transpose.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c types.c -o types.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c uniqlist.c -o uniqlist.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c utils.c -o utils.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c vecseq.c -o vecseq.o x86_64-conda_cos6-linux-gnu-cc -I"/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/include" -DNDEBUG -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -fpic -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/include -fdebug-prefix-map=/tmp/build/80754af9/r-base_1570124924484/work=/usr/local/src/conda/r-base-3.6.1 -fdebug-prefix-map=/azureml-envs/azureml_7930c16823057d9637678fa895a4db69=/usr/local/src/conda-prefix -c wrappers.c -o wrappers.o x86_64-conda_cos6-linux-gnu-cc -shared -L/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -L/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -Wl,-rpath-link,/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib -o data.table.so assign.o between.o bmerge.o chmatch.o cj.o coalesce.o dogroups.o fastmean.o fcast.o fifelse.o fmelt.o forder.o frank.o fread.o freadR.o froll.o frollR.o frolladaptive.o fsort.o fwrite.o fwriteR.o gsumm.o ijoin.o init.o inrange.o nafill.o nqrecreateindices.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o snprintf.o subset.o transpose.o types.o uniqlist.o utils.o vecseq.o wrappers.o -L/azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/lib -lR [91minstalling to /azureml-envs/azureml_7930c16823057d9637678fa895a4db69/lib/R/library/00LOCK-data.table/00new/data.table/libs [0m[91m** R [0m[91m** inst [0m[91m** byte-compile and prepare package for lazy loading [0m[91m** help [0m[91m*** installing help indices [0m[91m** building package indices [0m[91m** installing vignettes [0m[91m** testing if installed package can be loaded from temporary location [0mError: package or namespace load failed for ‘data.table’ in library.dynam(lib, package, package.lib): shared object ‘datatable.so’ not found Error: loading failed Execution halted

hermandr avatar Sep 18 '20 07:09 hermandr

Hi @hermandr the above error looks like the installation is failing for data.table because it has a system dependency. System dependencies cannot be installed through the cran_packages convenience parameter, you will have to create a custom docker image for your environment (see the vignette I linked previously)

mx-iao avatar Sep 24 '20 18:09 mx-iao

OK Noted thanks for your info

hermandr avatar Sep 25 '20 03:09 hermandr

Solved the xgboost installation, In the dockerfile need to have: RUN apt-get install -y pkg-config RUN R -e "install.packages(install.packages('xgboost',version='0.90.0.1', repos='https://cloud.r-project.org'))"

pkg-config takes care of make compile arguments for packages.

For creating custom docker instances, look at this post Replicating the Dockerfile produces errors

hermandr avatar Oct 23 '20 11:10 hermandr

Thanks for sharing your solution, @hermandr!

diondrapeck avatar Oct 26 '20 13:10 diondrapeck