netcdf-c
netcdf-c copied to clipboard
Unidata
NetCDF 4.9.0: undefined reference to `ncp_win32_api' on MinGW (libncpoco)
I tried to update the NetCDF version available to julia users. However, netCDF 4.9.0 fails to build during cross-compilation for x86_64-w64-mingw32 with the error "undefined reference to `ncp_win32_api'" .
- the version of the software with which you are encountering an issue
NetCDF 4.9.0
- environmental information (i.e. Operating System, compiler info, java version, python version, etc.)
target x86_64-w64-mingw32 (cross compiled from Linux) With gcc version: x86_64-w64-mingw32-gcc (GCC) 4.8.5
- a description of the issue with the steps needed to reproduce it
Here are the steps how we compile NetCDF (https://github.com/JuliaPackaging/Yggdrasil/blob/master/N/NetCDF/common.jl#L59)
./configure --prefix=/workspace/destdir --build=x86_64-linux-musl --host=x86_64-w64-mingw32 --enable-shared --disable-static --disable-dap-remote-tests --disable-testsets
make LDFLAGS="-L/workspace/destdir/bin -lhdf5-0 -lhdf5_hl-0 -lcurl-4 -lz -no-undefined -Wl,--export-all-symbols" -j8
The build step fails with:
[22:59:58] make[2]: Entering directory '/workspace/srcdir/netcdf-c-4.9.0/liblib'
[22:59:58] /bin/sh ../libtool --tag=CC --mode=compile cc -DHAVE_CONFIG_H -I. -I.. -I../include -I../include -I../oc2 -I../libnczarr -I../libhdf5 -I../libdap2 -I
../oc -I../libdap4 -I../libncxml -I../libsrc4 -I../libnczarr -I../libncpoco -I/workspace/destdir/include -std=c99 -fno-strict-aliasing -MT libnetcdf_la-nc_initialize
.lo -MD -MP -MF .deps/libnetcdf_la-nc_initialize.Tpo -c -o libnetcdf_la-nc_initialize.lo `test -f 'nc_initialize.c' || echo './'`nc_initialize.c
[22:59:58] libtool: compile: cc -DHAVE_CONFIG_H -I. -I.. -I../include -I../include -I../oc2 -I../libnczarr -I../libhdf5 -I../libdap2 -I../oc -I../libdap4 -I../libncx
ml -I../libsrc4 -I../libnczarr -I../libncpoco -I/workspace/destdir/include -std=c99 -fno-strict-aliasing -MT libnetcdf_la-nc_initialize.lo -MD -MP -MF .deps/libnetcdf
_la-nc_initialize.Tpo -c nc_initialize.c -DDLL_EXPORT -DPIC -o .libs/libnetcdf_la-nc_initialize.o
[22:59:58] mv -f .deps/libnetcdf_la-nc_initialize.Tpo .deps/libnetcdf_la-nc_initialize.Plo
[22:59:58] /bin/sh ../libtool --tag=CC --mode=link cc -std=c99 -fno-strict-aliasing -version-info 20:0:1 -L/workspace/destdir/bin -lhdf5-0 -lhdf5_hl-0 -lcurl-4 -
lz -no-undefined -Wl,--export-all-symbols -o libnetcdf.la -rpath /workspace/destdir/lib libnetcdf_la-nc_initialize.lo ../libdispatch/libnetcdf2.la ../libdispatch/libd
ispatch.la ../libsrc/libnetcdf3.la ../libhdf5/libnchdf5.la ../libdap2/libdap2.la ../oc2/liboc.la ../libdap4/libdap4.la ../libncxml/libncxml.la ../libsrc4/libnetcdf4
.la ../libnczarr/libnczarr.la ../libncpoco/libncpoco.la -lm -lsz -lxml2 -lhdf5-0 -lhdf5_hl-0 -lcurl-4 -lz
[22:59:58] libtool: link: cc -shared .libs/libnetcdf_la-nc_initialize.o -Wl,--whole-archive ../libdispatch/.libs/libnetcdf2.a ../libdispatch/.libs/libdispatch.a ../
libsrc/.libs/libnetcdf3.a ../libhdf5/.libs/libnchdf5.a ../libdap2/.libs/libdap2.a ../oc2/.libs/liboc.a ../libdap4/.libs/libdap4.a ../libncxml/.libs/libncxml.a ../libs
rc4/.libs/libnetcdf4.a ../libnczarr/.libs/libnczarr.a ../libncpoco/.libs/libncpoco.a -Wl,--no-whole-archive -L/workspace/destdir/bin -lsz -lxml2 -lhdf5-0 -lhdf5_hl-0
-lcurl-4 -lz -Wl,--export-all-symbols -o .libs/libnetcdf-19.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libnetcdf.dll.a
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x40): undefined reference to `ncp_win32_api'
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x4e): undefined reference to `ncp_win32_api'
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x5c): undefined reference to `ncp_win32_api'
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x6a): undefined reference to `ncp_win32_api'
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x78): undefined reference to `ncp_win32_api'
[22:59:58] ../libncpoco/.libs/libncpoco.a(ncpoco.o):ncpoco.c:(.text+0x86): more undefined references to `ncp_win32_api' follow
[22:59:58] /opt/x86_64-w64-mingw32/bin/../lib/gcc/x86_64-w64-mingw32/4.8.5/../../../../x86_64-w64-mingw32/bin/ld: ../libncpoco/.libs/libncpoco.a(ncpoco.o): bad reloc
address 0x0 in section `.pdata'
[22:59:58] collect2: error: ld returned 1 exit status
[22:59:58] make[2]: *** [Makefile:612: libnetcdf.la] Error 1
[22:59:58] make[2]: Leaving directory '/workspace/srcdir/netcdf-c-4.9.0/liblib'
[22:59:58] make[1]: *** [Makefile:764: all-recursive] Error 1
[22:59:58] make[1]: Leaving directory '/workspace/srcdir/netcdf-c-4.9.0'
[22:59:58] make: *** [Makefile:609: all] Error 2
[22:59:58] ---> make LDFLAGS="${LDFLAGS_MAKE}" -j${nproc}
[22:59:58] ---> make LDFLAGS="${LDFLAGS_MAKE}" -j${nproc}
[22:59:58] Previous command exited with 2
[22:59:58] Child Process exited, exit code 2
┌ Warning: Build failed, the following log files were generated:
│ - ${WORKSPACE}/srcdir/netcdf-c-4.9.0/config.log
Attached is the file config.log
Thank you for your help.
Maybe --disable-plugins is necessary for Windows?
https://github.com/Unidata/netcdf-c/blob/5df5539576c5b2aa8f31d4b50c4f8258925589dd/.github/workflows/run_tests_win_mingw.yml#L38
(the compilation succeeds with this option)
It looks like the file libncpoco/cp_win32.c is not being compiled. Can you check your build to see if it compiles that file?
Indeed, this file is not compiled:
$ LANG=C ls -la * .libs/
-rwxr-xr-x 1 abarth abarth 278 Jun 10 23:04 CMakeLists.txt
-rwxr-xr-x 1 abarth abarth 2043 Jun 10 23:04 COPYRIGHT
-rw-r--r-- 1 abarth abarth 23996 Jun 13 09:19 Makefile
-rwxr-xr-x 1 abarth abarth 1199 Jun 10 23:04 Makefile.am
-rw-rw-r-- 1 abarth abarth 25603 Jun 10 23:04 Makefile.in
-rw-rw-r-- 1 abarth abarth 4617 Jun 10 23:04 README.md
-rw-rw-r-- 1 abarth abarth 1401 Jun 10 23:04 SourceLicence
-rwxr-xr-x 1 abarth abarth 5354 Jun 10 23:04 cp_test.c
-rwxr-xr-x 1 abarth abarth 3689 Jun 10 23:04 cp_unix.c
-rw-r--r-- 1 abarth abarth 266 Jun 13 09:13 cp_unix.lo
-rwxr-xr-x 1 abarth abarth 3733 Jun 10 23:04 cp_win32.c
-rwxr-xr-x 1 abarth abarth 100 Jun 10 23:04 cptestlib.c
-rw-r--r-- 1 abarth abarth 923 Jun 13 09:13 libncpoco.la
-rwxr-xr-x 1 abarth abarth 2748 Jun 10 23:04 ncpoco.c
-rwxr-xr-x 1 abarth abarth 2327 Jun 10 23:04 ncpoco.h
-rw-r--r-- 1 abarth abarth 264 Jun 13 09:13 ncpoco.lo
.libs/:
total 24
drwxr-xr-x 2 abarth abarth 4096 Jun 13 09:13 .
drwxrwxr-x 4 abarth abarth 4096 Jun 13 09:19 ..
-rw-r--r-- 1 abarth abarth 2620 Jun 13 09:13 cp_unix.o
-rw-r--r-- 1 abarth abarth 5100 Jun 13 09:13 libncpoco.a
lrwxrwxrwx 1 abarth abarth 15 Jun 13 09:13 libncpoco.la -> ../libncpoco.la
-rw-r--r-- 1 abarth abarth 2092 Jun 13 09:13 ncpoco.o
Maybe the build system is confused since we do a cross-compiliation (as cp_unix.o is present).
Ok, the problem is probably the conditional code in libncpoco/Makefile.am. It currently read as follows:
if ! ISMSVC
if ! ISMINGW
libncpoco_la_SOURCES += cp_unix.c
else
libncpoco_la_SOURCES += cp_win32.c
endif
else
libncpoco_la_SOURCES += cp_win32.c
endif
Btw, the value of ISMSVC and ISMINGW comes from this code in configure.ac
case "`uname`" in
CYGWIN*) ISCYGWIN=yes;;
Darwin*) ISOSX=yes;;
WIN*) ISMSVC=yes;;
MINGW*) ISMINGW=yes;;
esac
For some reason, this is not working. Does anything stand out to you?
I believe you are correct that the cross compile is the problem. Not sure how to fix the above tests to get this correct.
Inside the docker container from the Julia build system, I run these tests on a bash shell:
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $target # probably julia specific
x86_64-w64-mingw32
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # uname -a
MSYS_NT-6.3 gher17 2.8.2(0.313/5/3) Wed Feb 13 00:49:00 UTC 2013 x86_64 Cygwin
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # case "`uname`" in
CYGWIN*) ISCYGWIN=yes;;
Darwin*) ISOSX=yes;;
WIN*) ISMSVC=yes;;
MINGW*) ISMINGW=yes;;
esac
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $ISCYGWIN
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $ISOSX
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $ISMSVC
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # echo $ISMINGW
sandbox:${WORKSPACE}/srcdir/netcdf-c-4.9.0 # uname
MSYS_NT-6.3
Maybe we need to test: MSYS*) ISMINGW=yes;; Or there could be a source file e.g. cp_platform.c which would conditionally include cp_unix.c or cp_win32.c
#ifdef _WIN32
#include "cp_win32.c"
#else
#include "cp_unix.c"
#endif
@Alexander-Barth Out of curiosity, is the Docker container you mentioned one that could be run/used outside of the Julia packaging system? I'm digging into the README.md and want to make sure I don't dive too deep into this without knowing if it's leading anywhere. But the idea of a Docker image can generate cross-platform netCDF packages is interesting to me, and I'd love to take a look at what you're using, if it is possible to in a stand-alone fashion. FWIW, I have a good bit of familiarity with Docker. Thanks!
@WardF The build environment (https://binarybuilder.org/) is in fact a julia package which setups the container and the compiler toolchain and it uses the docker image from alpine linux as a base. While on Windows and Mac OS, the docker engine is also used, it seem that on Linux a more lightweight and custom sandbox runtime is used.
There is more technical information here: https://docs.binarybuilder.org/dev/rootfs/
The resulting binaries (libraries, executable...) are in fact independent of julia and are just simple tarballs (one per architecture / OS ).
I don't know too much about the internals (and maybe @staticfloat can correct me if I misunderstood something), but as a user a lot of things (handling the build chain, injecting software dependencies in the container) happen automatically. If a build fails with, the --debug option, allows one to automatically drops into a shell. Per default BinaryBuilder also keeps track (log files, sources) of previous attempts. For new packages, there is also a command line Wizard were you can specify your dependencies and supported platforms. The main entry point as a developer are the build_tarballs.jl julia script which essentially contain a shell script to build the binaries, e.g.
https://github.com/JuliaPackaging/Yggdrasil/blob/master/U/UDUNITS/build_tarballs.jl
Resulting binaries:
https://github.com/JuliaBinaryWrappers/UDUNITS_jll.jl/releases/tag/UDUNITS-v2.2.28%2B0
For NetCDF, it is a bit more complicated as there are shared dependencies (libcurl among others) that need to be synchronized between Julia and NetCDF:
https://github.com/JuliaPackaging/Yggdrasil/tree/master/N/NetCDF https://github.com/JuliaBinaryWrappers/NetCDF_jll.jl/releases/tag/NetCDF-v400.802.103%2B0
(sadly the platform support of NetCDF is not nearly as large as UDUNITS, but this is mostly due to the fact that HDF5 does not support cross-compilation and some Windows issues)
Interesting, thank you! That is worth reading through and educating myself; I am not super familiar with Julia, but I see the opportunity to extract chunks of technologies I am much more familiar with. If nothing else, it is a really interesting toolchain to parse, and I quite like the idea of being able to mint cross-platform libraries using docker. Thanks for the comprehensive introduction!
Btw, the value of
ISMSVCandISMINGWcomes from this code in configure.achttps://github.com/Unidata/netcdf-c/blob/26c558203f4be1d29ec74b85151fbe487777719a/configure.ac#L105-L111
For some reason, this is not working. Does anything stand out to you?
This seems to be the kind of thing the --host option to ./configure is used for; using AC_CANONICAL_HOST gets a standard value, there are suggestions for using that value here, which suggest something like this:
AC_CANONICAL_HOST
AS_CASE([$host],
[*-*-cygwin], [ISCYGWIN=yes],
[*-*-darwin*], [ISOSX=yes],
[*-*-mingw*], [ISMINGW=yes],
[*-*-msys], [ISMINGW=yes],
[*-*-win*], [ISMSVC=yes],
[]
)
using a macro from here and the guesses from here. For some reason, GNU Autoconf doesn't seem to have code for guessing MicroSoft Visual C, so compiling using MSVC and the Autotools build scripts (./configure && make) will require specifying --build and/or --host or some fallback code to catch the failure to guess the build system and set the variables as in the current code.
(I think one of automake or libtool already pulls in AC_CANONICAL_HOST, so this wouldn't add new code. --host and friends will recognize winnt, windows, wince, and certain variations of those if specified on the command line, configure just won't guess those on its own.)
I made this change on my fork, and have seen no new failures on Cygwin or MinGW.
