NCDatasets.jl
NCDatasets.jl copied to clipboard
Alexander-Barth
Broader platform support
It would be great if NCDatasets.jl could be run on all platforms on which julia is supported. However, currently only on these platforms we have a NetCDF binary:
NetCDF.v400.702.400.aarch64-linux-gnu.tar.gz
NetCDF.v400.702.400.i686-w64-mingw32.tar.gz
NetCDF.v400.702.400.x86_64-apple-darwin.tar.gz
NetCDF.v400.702.400.x86_64-linux-gnu.tar.gz
NetCDF.v400.702.400.x86_64-w64-mingw32.tar.gz
Installation of NCDatasets.jl will fail on other platforms. Notably, this leaves out Raspberry PIs (32-bit armv7l), Linux on PowerPC, FreeBSD, Linux with musl, Apple M1,...
To support more architectures the NetCDF_jll package and HDF5_jll package would need to be updated. As a first test, it would be useful to know if native compilation (of the NetCDF sources) on these platforms pass all the tests (make check) and then attempt cross-compilation (using BinaryBuilder).
ref https://github.com/Alexander-Barth/NCDatasets.jl/issues/116
Regarding cross-compilation, after tracing the breadcrumbs again it seems we now have an upstream issue we can refer to: https://github.com/HDFGroup/hdf5/issues/1203.
I'm very interested having NCDatasets.jl support Apple Silicon. I just installed netcdf 4.8.1 and HDF5 1.12.1 via homebrew on my M1 mac and they seem to function fine without having done too deep testing.
https://formulae.brew.sh/formula/hdf5 https://formulae.brew.sh/formula/netcdf
The first step would be to make a PR to update these build instructions to support Apple M1 (or any new architecture):
https://github.com/JuliaPackaging/Yggdrasil/blob/master/H/HDF5/build_tarballs.jl
So far, it uses HDF5 from python on MacOS x86_64.
I proposed some changes, but I don't know how to test whether they work.
One should be able to test it with this command (https://docs.binarybuilder.org/stable/):
julia --color=yes build_tarballs.jl --verbose
You would need to install docker. Maybe you need to run this using the x86_64 emulator. It should create a tar.gz file for your architecture. When you decompress it, it should contain the library for your system.
Maybe this is helpful to force Julia to use a custom library (for HDF5 and NetCDF):
https://github.com/Alexander-Barth/NCDatasets.jl/issues/126#issuecomment-818045715
In my miniforge lib directory I saw the following two netcdf library files: libnetcdf.19.dylib and libnetcdf.a. Would either of them work? I also tried to follow the steps you mentioned for forcing Julia to use custom library. When I tried it with libnetcdf.19.dylib, I'm still getting libnetcdf not defined error when using NCDatasets. Is there anything extra I need to do to make NCDatasets see the change in libnetcdf?
In Linux libnetcdf.a, is a static library (which will not work, as we need a dynamic library). I guess this is true of also for Mac OS. So I would go with libnetcdf.19.dylib. Did you restart the julia session after calling set_preferences! ? Maybe you can include all the steps that lead to the error with the full error message as a comment here?
The libraries on which netcdf depends (HDF5, zlib, ...) can also be a source of problem. The promise of Yggdrasil is to compile/prepare all the libraries so ensure their comparability otherwise there are problems to similar to the issue described here: https://github.com/JuliaIO/ImageMagick.jl/issues/140
Related issue for reference: https://github.com/h5py/h5py/issues/2008
With the latest NetCDF_jll version (v400.702.402+0) and HDF5_jll (v1.12.1+0), I'm able to use NCDatasets.jl on my Apple Silicon M1 mac without the libnetcdf not defined error.
With the latest NetCDF_jll version (v400.702.402+0) and HDF5_jll (v1.12.1+0), I'm able to use NCDatasets.jl on my Apple Silicon M1 mac without the libnetcdf not defined error.
This is great to know! Thanks a lot @felixcremer for implementing this!
https://github.com/JuliaPackaging/Yggdrasil/pull/4481/files
Hi. I'm having some trouble getting NCDatasets to work on my M1 MacBook air. When I add NCDatasets it does not use the latest versions of NETCDF_jll and HDF5_jll, which I guess is the problem. But when I try to force it to use these versions (NetCDF_jll v400.702.402+0 and HDF5_jll v1.12.1+0) I get errors that the versions are not compatible with what NCDatasets is looking for. Any help? Below is output from my REPL. I am using julia v1.7.2 (2022-02-06).
(experiments) pkg> add NCDatasets
Resolving package versions...
Updating `~/Research/MixedLayerModel.jl/experiments/Project.toml`
[85f8d34a] + NCDatasets v0.12.4
Updating `~/Research/MixedLayerModel.jl/experiments/Manifest.toml`
[179af706] + CFTime v0.1.2
[85f8d34a] + NCDatasets v0.12.4
[0234f1f7] + HDF5_jll v1.12.0+1
[7243133f] + NetCDF_jll v400.702.400+0
As a test (or work-around) can you install version 0.11.9 of NCDatasets?
]add [email protected]
Does this work?
It is kind of a struggle right now to support Windows and Mac OS M1 as there is no NetCDF_jll version which runs on both and dependencies must be platform independent for Julia's package manager. The real fix would be to make the latest version of NetCDF_jll available on Windows.
As a test (or work-around) can you install version 0.11.9 of NCDatasets?
]add [email protected]Does this work?
Yes thank you, this does work! I hope the platform support of NetCDF_jll is improved in the future so this isn't so annoying for you to maintain!
Not sure if this is related but seems like NCDatasets can't be used on Linux with x86 architecture + Julia 1.8. Am I right?
navidcon@satori-login-001:~/test_project$ julia --project
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.8.2 (2022-09-29)
_/ |\__'_|_|_|\__'_| |
|__/ |
(@v1.8) pkg> activate .
Activating new project at `~/test_project`
(test_project) pkg> add NCDatasets
Updating registry at `~/.julia/registries/General.toml`
Resolving package versions...
Updating `~/test_project/Project.toml`
[85f8d34a] + NCDatasets v0.12.8
Updating `~/test_project/Manifest.toml`
[179af706] + CFTime v0.1.2
[34da2185] + Compat v4.3.0
[864edb3b] + DataStructures v0.18.13
[692b3bcd] + JLLWrappers v1.4.1
[85f8d34a] + NCDatasets v0.12.8
[bac558e1] + OrderedCollections v1.4.1
[21216c6a] + Preferences v1.3.0
[0234f1f7] + HDF5_jll v1.12.2+2
[94ce4f54] + Libiconv_jll v1.16.1+1
[7243133f] + NetCDF_jll v400.902.5+1
[458c3c95] + OpenSSL_jll v1.1.17+0
[02c8fc9c] + XML2_jll v2.9.14+0
[0dad84c5] + ArgTools v1.1.1
[56f22d72] + Artifacts
[2a0f44e3] + Base64
[ade2ca70] + Dates
[f43a241f] + Downloads v1.6.0
[7b1f6079] + FileWatching
[b77e0a4c] + InteractiveUtils
[b27032c2] + LibCURL v0.6.3
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[ca575930] + NetworkOptions v1.2.0
[44cfe95a] + Pkg v1.8.0
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA v0.7.0
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[fa267f1f] + TOML v1.0.0
[a4e569a6] + Tar v1.10.1
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
[e66e0078] + CompilerSupportLibraries_jll v0.5.2+0
[deac9b47] + LibCURL_jll v7.84.0+0
[29816b5a] + LibSSH2_jll v1.10.2+0
[c8ffd9c3] + MbedTLS_jll v2.28.0+0
[14a3606d] + MozillaCACerts_jll v2022.2.1
[4536629a] + OpenBLAS_jll v0.3.20+0
[83775a58] + Zlib_jll v1.2.12+3
[8e850b90] + libblastrampoline_jll v5.1.1+0
[8e850ede] + nghttp2_jll v1.48.0+0
[3f19e933] + p7zip_jll v17.4.0+0
julia> using NCDatasets
ERROR: InitError: UndefVarError: libnetcdf not defined
Stacktrace:
[1] nc_inq_libvers
@ ~/.julia/packages/NCDatasets/ipGBH/src/netcdf_c.jl:242 [inlined]
[2] netcdf_version()
@ NCDatasets ~/.julia/packages/NCDatasets/ipGBH/src/netcdf_c.jl:2157
[3] init_certificate_authority()
@ NCDatasets ~/.julia/packages/NCDatasets/ipGBH/src/netcdf_c.jl:2170
[4] __init__()
@ NCDatasets ~/.julia/packages/NCDatasets/ipGBH/src/NCDatasets.jl:33
[5] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
@ Base ./loading.jl:831
[6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
@ Base ./loading.jl:1039
[7] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1315
[8] _require_prelocked(uuidkey::Base.PkgId)
@ Base ./loading.jl:1200
[9] macro expansion
@ ./loading.jl:1180 [inlined]
[10] macro expansion
@ ./lock.jl:223 [inlined]
[11] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1144
during initialization of module NCDatasets
(test_project) pkg> st
Status `~/test_project/Project.toml`
[85f8d34a] NCDatasets v0.12.8
julia> versioninfo()
Julia Version 1.8.2
Commit 36034abf26 (2022-09-29 15:21 UTC)
Platform Info:
OS: Linux (ppc64le-redhat-linux)
CPU: 128 × POWER9, altivec supported
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, pwr9)
Threads: 24 on 128 virtual cores
Environment:
LD_LIBRARY_PATH = /software/cuda/11.4/lib:/software/cuda/11.4/lib64
JULIA_DEPOT_PATH = /home/navidcon/.julia:/share/julia/site/
JULIA_CUDA_USE_BINARYBUILDER = false
JULIA_LOAD_PATH = @:@v#.#:@stdlib:@site
JULIA_NUM_THREADS = 24
julia> Sys.ARCH
:powerpc64le
Julia>
Yes, this is the PowerPC architecture (x86 architecture referrers generally to the Intel/AMD CPU architecture, which is supported in its 64-bit variant). As a first step, we would need to repack HDF5 for PowerPC which seems to be available (https://anaconda.org/anaconda/hdf5/files). Can somebody interested in PowerPC make a PR to HDF5_jll ?
The supported platforms of NetCDF_jll has been recently significantly extended:
https://github.com/JuliaBinaryWrappers/NetCDF_jll.jl/releases/tag/NetCDF-v400.902.208%2B0
I am closing this issue as the platform support has significantly improved thanks to these changes in HDF5: https://github.com/JuliaPackaging/Yggdrasil/commit/bf78a45f9a781bf726d2cf48ab23ea42324e4515