easybuilders
R installation doesn't fetch required CRAN packages
I am hoping this is an easy one.
When trying to install R:
[easybuild@hyalite R]$ eb R-3.3.1-foss-2016a.eb --robot
It fails with this message:
== FAILED: Installation ended unsuccessfully (build directory: /cm/shared/hyalite/apps/eb/build/R/3.3.1/foss-2016a): build fail
ed (first 300 chars): Couldn't find file magic_1.5-6.tar.gz anywhere, and downloading it didn't work either... Paths attempted
(in order): /cm/shared/hyalite/apps/eb/EasyBuild/3.1.2/lib/python2.6/site-packages/easybuild_easyconfigs-3.1.2-py2.6.egg/easybu
ild/easyconfigs/r/R/r/R/extensions/magic_1.5-6.tar.gz, /cm/shared/hy
== Results of the build can be found in the log file(s) /tmp/eb-8HKD6z/easybuild-R-3.3.1-20170620.115548.syDpI.log
ERROR: Build of /cm/shared/hyalite/apps/eb/EasyBuild/3.1.2/lib/python2.6/site-packages/easybuild_easyconfigs-3.1.2-py2.6.egg/ea
sybuild/easyconfigs/r/R/R-3.3.1-foss-2016a.eb failed (err: "build failed (first 300 chars): Couldn't find file magic_1.5-6.tar.
gz anywhere, and downloading it didn't work either... Paths attempted (in order): /cm/shared/hyalite/apps/eb/EasyBuild/3.1.2/li
b/python2.6/site-packages/easybuild_easyconfigs-3.1.2-py2.6.egg/easybuild/easyconfigs/r/R/r/R/extensions/magic_1.5-6.tar.gz, /c
m/shared/hy")
At first I figured that this was due to an accidental mistake, so I went and found magic_1.5-6.tar.gz, downloaded it into /cm/shared/hyalite/apps/eb/EasyBuild/3.1.2/li b/python2.6/site-packages/easybuild_easyconfigs-3.1.2-py2.6.egg/easybuild/easyconfigs/r/R/r/R/extensions/ and then the installation failed on this:
== FAILED: Installation ended unsuccessfully (build directory: /cm/shared/hyalite/apps/eb/build/R/3.3.1/foss-2016a): build fail
ed (first 300 chars): Couldn't find file geometry_0.3-6.tar.gz anywhere, and downloading it didn't work either... Paths attempt
ed (in order): /cm/shared/hyalite/apps/eb/EasyBuild/3.1.2/lib/python2.6/site-packages/easybuild_easyconfigs-3.1.2-py2.6.egg/eas
ybuild/easyconfigs/r/R/r/R/extensions/geometry_0.3-6.tar.gz, /cm/sha
== Results of the build can be found in the log file(s) /tmp/eb-u4KWKa/easybuild-R-3.3.1-20170620.120252.ZgRNF.log
ERROR: Build of /cm/shared/hyalite/apps/eb/EasyBuild/3.1.2/lib/python2.6/site-packages/easybuild_easyconfigs-3.1.2-py2.6.egg/ea
sybuild/easyconfigs/r/R/R-3.3.1-foss-2016a.eb failed (err: "build failed (first 300 chars): Couldn't find file geometry_0.3-6.t
ar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /cm/shared/hyalite/apps/eb/EasyBuild/3.1.2
/lib/python2.6/site-packages/easybuild_easyconfigs-3.1.2-py2.6.egg/easybuild/easyconfigs/r/R/r/R/extensions/geometry_0.3-6.tar.
gz, /cm/sha")
I wanted to see the scope of what was missing, so I edited the easybuild recipe:
[easybuild@hyalite R]$ vim /cm/shared/hyalite/apps/eb/EasyBuild/3.1.2/lib/python2.6/site-packages/easybuild_easyconfigs-3.1.2-p
y2.6.egg/easybuild/easyconfigs/r/R/R-3.3.1-foss-2016a.eb
And found that the paths looked just fine:
ext_options = {
'source_urls': [
'http://cran.r-project.org/src/contrib/Archive/%(name)s', # package archive
'http://cran.r-project.org/src/contrib/', # current version of packages
'http://cran.freestatistics.org/src/contrib', # mirror alternative for current packages
],
'source_tmpl': name_tmpl,
}
I checked DNS, I can easily download the required geometry_0.3-6.tar.gz from the second URL in source_urls from the system.
The list of required packages is very long, so downloading them manually will be kind of a pain, is there some other way to resolve this?
Oh, also:
[easybuild@hyalite R]$ eb --version
This is EasyBuild 3.1.2 (framework: 3.1.2, easyblocks: 3.1.2) on host hyalite.
I found the problem. I ran in debug mode, and found this:
== 2017-06-20 12:23:19,485 easyblock.py:652 DEBUG Trying to download file filehash_2.3.tar.gz from http://cran.r-project.org/src/contrib/Archive/filehash/filehash_2.3.tar.gz to /cm/shared/hyalite/apps/eb/sources/r/R/extensions/filehash_2.3.tar.gz ...
== 2017-06-20 12:23:19,485 filetools.py:349 DEBUG Trying to download filehash_2.3.tar.gz from http://cran.r-project.org/src/contrib/Archive/filehash/filehash_2.3.tar.gz to /cm/shared/hyalite/apps/eb/sources/r/R/extensions/filehash_2.3.tar.gz
== 2017-06-20 12:23:19,485 filetools.py:356 DEBUG Using timeout of 10 seconds for initiating download
== 2017-06-20 12:23:19,485 filetools.py:1025 DEBUG Not creating existing path /cm/shared/hyalite/apps/eb/sources/r/R/extensions
== 2017-06-20 12:23:20,069 filetools.py:374 DEBUG response code for given url http://cran.r-project.org/src/contrib/Archive/filehash/filehash_2.3.tar.gz: 200
== 2017-06-20 12:23:30,517 filetools.py:387 WARNING IOError occurred while trying to download http://cran.r-project.org/src/contrib/Archive/filehash/filehash_2.3.tar.gz to /cm/shared/hyalite/apps/eb/sources/r/R/extensions/filehash_2.3.tar.gz: timed out
Then I tried using wget to grab the URL for filehash. It did not complete, it got to ~140kb and then just hung. However, the microsoft provided repository seems reliable, so I edited the source_urls to this:
ext_options = {
'source_urls': [
'https://mran.microsoft.com/src/contrib/Archive/%(name)s', # microsoft package archive
'https://mran.microsoft.com/src/contrib/', # microsoft current version of packages
'http://cran.r-project.org/src/contrib/Archive/%(name)s', # package archive
'http://cran.r-project.org/src/contrib/', # current version of packages
'http://cran.freestatistics.org/src/contrib', # mirror alternative for current packages
],
'source_tmpl': name_tmpl,
}
And then it worked. The CRAN repository doesn't appear to be 100% reliable.
But it's not the fault of easybuild. Still, maybe keep this issue around for others who run into this problem?
@pol This could be a temporary issue with CRAN, it's the first time I've seen this reported.
Maybe using --download-timeout with eb helps? Default is 10 seconds, maybe try with 100s or longer?
We could make the warning more verbose, i.e. let it pop up in the output rather than just in the log? Not sure how often this sort of things occurs though...
I don't know, I waited a while for the wget to complete, and I ended up ctrl-c'ing the process. I am not sure what's going on, but the microsoft packages downloaded without any timeout problem.
I do notice that there's no mirror for archive packages, so maybe adding this to the source_urls would be sufficient:
'http://cran.freestatistics.org/src/contrib/Archive/%(name)s', # mirror alternative for archive packages
If, for whatever reason, you are uncomfortable with including the microsoft mran repository. AFAIK mran is totally CRAN compatible, but I could be wrong.
After a bit of digging, I think that the mran repository ultimately resolves to https://cran.revolutionanalytics.com, which just looks like a CRAN mirror. It could be included as an alternative to the freestatistics.org repository if you don't want "microsoft.com" in the URL.