HDFS.jl
HDFS.jl copied to clipboard
JuliaParallel
[PkgEval] HDFS may have a testing issue on Julia 0.3 (2015-04-04)
PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their tests (if available) on both the stable version of Julia (0.3) and the nightly build of the unstable version (0.4). The results of this script are used to generate a package listing enhanced with testing results.
On Julia 0.3
- On 2015-04-02 the testing status was
No tests, but package loads. - On 2015-04-04 the testing status changed to
Package doesn't load.
No tests, but package loads. means that PackageEvaluator did not find tests for your package. However, trying to load your package with using worked.
Package doesn't load. means that PackageEvaluator did not find tests for your package. Additionally, trying to load your package with using failed.
This issue was filed because your testing status became worse. No additional issues will be filed if your package remains in this state, and no issue will be filed if it improves. If you'd like to opt-out of these status-change messages, reply to this message saying you'd like to and @IainNZ will add an exception. If you'd like to discuss PackageEvaluator.jl please file an issue at the repository. For example, your package may be untestable on the test machine due to a dependency - an exception can be added.
Test log:
>>> 'Pkg.add("HDFS")' log
INFO: Cloning cache of HDFS from git://github.com/JuliaParallel/HDFS.jl.git
INFO: Installing HDFS v0.1.0
INFO: Package database updated
>>> 'using HDFS' log
Julia Version 0.3.6
Commit 0c24dca (2015-02-17 22:12 UTC)
Platform Info:
System: Linux (x86_64-unknown-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Nehalem)
LAPACK: libopenblas
LIBM: libopenlibm
LLVM: libLLVM-3.3
/home/vagrant/testpkg/v0.3/HDFS
ERROR: could not load module libhdfs: libhdfs: cannot open shared object file: No such file or directory
in dlopen at c.jl:19
in include at ./boot.jl:245
in include_from_node1 at ./loading.jl:128
in reload_path at loading.jl:152
in _require at loading.jl:67
in require at loading.jl:51
in include at ./boot.jl:245
in include_from_node1 at loading.jl:128
in process_options at ./client.jl:285
in _start at ./client.jl:354
while loading /home/vagrant/testpkg/v0.3/HDFS/src/HDFS.jl, in expression starting on line 35
while loading /vagrant/releaseAL/PKGEVAL_HDFS_using.jl, in expression starting on line 4
>>> test log
no tests to run
>>> end of log
HDFS.jl now requires libhdfs.so and JVM shared objects during package load. Looks like JVM is already available in PackageEvaluator.jl.
So, libhdfs.so is the only requirement which can be got from a Hadoop binary download. But I'm not sure if we should have the runtests.jl download it each time, or whether we can have it included in PackageEvaluator. I also have another Hadoop related package coming up which may require a similar treatment.
@IainNZ What do you suggest? Otherwise, can we add an exception for HDFS.jl?
How big is the download? If its not bad, then should just be a matter of adding a line or two https://github.com/IainNZ/PackageEvaluator.jl/blob/master/scripts/setup.sh right?
Is there any chance you could try it out in the Vagrant VM and estimate how long that adds (I'm guessing not long) and whether the tests run in the memory constraints of the VM?
@IainNZ The VM memory allocated does seem to be small, but it may be fine for some simple tests. I don't think it will add too much time to the tests. Will give it a try.
@IainNZ It seems that the instructions to install the Oracle JVM https://github.com/IainNZ/PackageEvaluator.jl/blob/b6ca87ced5c94d8636ea9b64078edc40ee5e4e0e/scripts/setup.sh#L79 do not work anymore.
Do you see that in your setup too?
Any objections to using OpenJDK 7 instead with apt-get install openjdk-7-jdk?
@IainNZ The basic tests seem to work fine in the VM. The modified setup.sh I used is here: https://gist.github.com/tanmaykm/c3e080421a2ffdf1ea59
Oh wow, that must have been a recent change, it was working a couple of days ago. Do you know if JavaCall etc work with OpenJDK 7 (@aviks ?). That setup script is kinda long, maybe the PR could put that in a separate file
Yes, that can go as a separate script.
@aviks, JavaCall tests failed here, but fixing that still results in NullPointerException error further down.
@tanmaykm I just updated pkg.julialang with fresh results, JavaCall and Taro seem to be working fine, so I'm not sure why it didn't work for you.
Strange, I did not find java installed in the VM when I built it, and issuing the apt-get commands threw an error. Let me try it again.
I actually ran runvagrant.sh, slightly modified to run only on release.
It's at /home/tanmaykm/test/PackageEvaluator.jl on julia.mit.edu.
It's still running, but I can already see the error: ==> release: E: Unable to locate package oracle-java7-installer in release.out.txt
@tanmaykm : you need to run
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
before the apt-get install oracle-java... line will work. Those above invocations probably go in the base image.
I am now running the unmodified setup.sh. I suppose this was already being done?
Ah! I see that python-software-properties is not being installed in the latest setup.sh.
@IainNZ Are the files on github the latest?
Yes. I'm not making any claim about setup.sh working on a given machine, its just designed to work on a clean install of Ubuntu 12.04, which it definitely does.
Ah ok... But when a new vagrant box is being setup, doesn't setup.sh run on a clean Ubuntu 12.04 anyway? Then maybe the difference is due to recent changes in the base ubuntu/precise64 image?
I could get java installed by including python-software-properties as @aviks suggested. I'll fetch ubuntu/precise64 afresh and try. And if it still requires python-software-properties, will add that to the changes in the PR. Does that sound good @IainNZ ?