Clarification re:zlib

[MichaelChirico]

data.table recently added support for emitting gzipped files, which requires zlib headers to build properly.

I recently happened to be using rocker/r-ver:3.5.3 to chase down a version-specific test failure and also noticed that there doesn't appear to be zlib support on this image:

docker run -it rocker/r-ver:3.5.3
install.packages('data.table', type = 'source', repos = 'http://Rdatatable.github.io/data.table')
# <suppressed>
# gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fwrite.c -o fwrite.o
# fwrite.c:22:48: fatal error: zlib.h: No such file or directory
 #include "zlib.h"      // for writing gzip file

And in fact zlib isn't found:

grep('zlib', list.files('/usr/local/include', recursive = TRUE), value=TRUE)
# character(0)
grep('zlib', list.files('/usr/local/lib/R/include', recursive = TRUE), value=TRUE)
# character(0)
grep('zlib', list.files('/usr/include', recursive = TRUE), value=TRUE)
# character(0)

This is easily solved e.g. with apt-get update && apt-get install zlib1g-dev; what drew my attention is Philippe's (original gzip support author's) pointing out that system zlib is required for R since 3.3.0 (with a version of zlib included in R sources before that):

https://github.com/Rdatatable/data.table/issues/3872#issuecomment-533578596

Was it an intentional design decision to omit this from the image (e.g. with minimalist constraints in mind)?

A bit more broadly, this raised flags for us that we might be increasing the dependency load of data.table with this new requirement (which in practice is really only a "suggested" dependency for users seeking support for compressed fwrite output), and all the more troublingly so that it's a system requirement (which are a lot more painful for inexperienced users to figure out since they can't install.packages their way out), so we're hoping any insights you offer can illuminate whether we're likely to encounter many users with a setup like that in r-ver:3.5.3 "in the wild" once the zlib-required version of data.table hits CRAN

[cboettig]

Thanks for the bug report, it's great to get these details!

A few quick points for clarification:

• zlib is included on the image: https://github.com/rocker-org/rocker-versioned/blob/master/r-ver/3.5.3/Dockerfile#L41

Note that your base R on this image zlib support, e.g. see ?gzcon()

• The zlib developer libraries are a build dependency for R, but not a runtime dependency. We install all the build dependencies to build R: https://github.com/rocker-org/rocker-versioned/blob/master/r-ver/Dockerfile#L71, but we remove them to save about half the image size.

Note that many packages, like readr, support creating gzipped data out of the box because they bind through the gzcon method in base R. If you want to install additional packages from source that need developer libraries/headers to compile that support, you're correct, you'll need to apt-get install those yourself.

Hope this helps. That said, we may indeed want/need to put zlib-dev libraries in one of the later images like tidyverse to streamline the data.table install.... Further discussion welcome!

[MichaelChirico]

thanks for the response! perfectly understood.

re downstream images, something I've thought about recently is something like a sys-deps folder of executables to install system dependencies for packages...

e.g. you could just do sys-deps/data.table to install data.table system dependencies. the problem is making it scalable -- it's a bit tough to track down system dependencies for R packages in an automated way in my experience

[cboettig]

@MichaelChirico That's a really great idea.

One thing we could do is support/document how to have a generic bash script get run as root prior to RStudio startup; e.g. if your docker run command links a volume with a apt.sh script, for instance.

Going a step beyond and automating the sys.deps for an individual package is harder, though a few folks have looked at using the metadata provided by Gabor's r-hub to do that. @nuest 's containerit https://github.com/o2r-project/containerit is one example. (ultimately that can founder a bit particularly in cases of older Debian or ubuntu releases where the version packaged by the distro isn't new enough)

[eddelbuettel]

[ Or turn it upside down and based on resolved dependency within a distro? Michael Rutter has over 4000 CRAN packages in his PPA -- if one uses the r-ubuntu nothing else is needed. Many approaches work. The metadata was part of cran2deb, it sort lives on (unexported) in the PPA work. If only we all had a bit more time... ]