GMT

[boriskaus]

This is a first attempt at creating a BinaryBuilder version of GMT, the Generic Mapping Tools. Doesn't work on windows because (among others) the current version of NetCDF isn't compiled for that.

[boriskaus]

The errors on windows are:

[21:27:19] CMakeFiles/gmtlib.dir/objects.a(gmt_fft.c.obj):gmt_fft.c:(.text+0x3825): undefined reference to `__imp_gethostname'
[21:27:19] CMakeFiles/gmtlib.dir/objects.a(gmt_init.c.obj):gmt_init.c:(.text+0x3d248): undefined reference to `__imp__set_fmode'
[21:27:19] collect2: error: ld returned 1 exit status
[21:27:19] make[2]: *** [src/CMakeFiles/gmtlib.dir/build.make:2561: src/gmt.dll] Error 1

if that rings a bell for someone, suggestions are helpful!

[giordano]

For gethostname I guess you need to link to -lws2_32? Not sure about _set_fmode, documentation doesn't indicate what you need to link to :confused:

[giordano]

According to

julia --compile=min -e 'using BinaryBuilderBase; BinaryBuilderBase.runshell(Platform("x86_64", "windows"))'
sandbox:${WORKSPACE} # for file in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/*.a; do nm $file | grep __imp__set_fmode && echo "found in ${file}"; done
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr100.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr110.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr120.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr120_app.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr120d.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr90.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libmsvcr90d.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libucrt.a
0000000000000000 I __imp__set_fmode
found in /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/lib/libucrtbase.a

_set_fmode is provided by a bunch of internal libraries, I'm not sure you're supposed to manually link to any of them.

[boriskaus]

thanks. If I manually compile this with

-lws2_32 -lmsvcr100

it works & the whole code compiles. Do you happen to know how I can tell CMake to take this libraries into account as well?

[giordano]

You have to add the libraries names to the target which needs them, for example with target_link_libraries or whatever they're using to specify the dependencies. ws2_32 is definitely public API, but msvcr100 is something very internal (like all other libraries which provide _set_fmode), you just picked up a random library from the list I shared, that doesn't sound right. I have no idea how _set_fmode is supposed to be properly linked.

[giordano]

Some creative uses of linker flags:

[21:27:01] /opt/bin/aarch64-linux-gnu-libgfortran4-cxx11/aarch64-linux-gnu-gcc [...] -Wl,-rpath,/usr/local/lib::::::::::::::::::::::: [...]
[...]
[21:27:04] /opt/bin/aarch64-linux-gnu-libgfortran4-cxx11/aarch64-linux-gnu-gcc [...] -Wl,-rpath,/usr/local/lib:/workspace/srcdir/gmt-6.2.0/build/src: [...] 

Linking a source directory?!?!?

[boriskaus]

ok, I keep being confused with CMake. If I manually add this at the end:

/opt/bin/x86_64-w64-mingw32-libgfortran4-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -std=gnu99  -shared -o gmt.dll -Wl,--out-implib,gmt.dll.a -Wl,--major-image-version,6,--minor-image-version,2 -Wl,--whole-archive CMakeFiles/gmtlib.dir/objects.a -Wl,--no-whole-archive @CMakeFiles/gmtlib.dir/linklibs.rsp -lws2_32 -lmsvcr100

it works. Yet, any attempt to add the libraries as flags results in something like

/opt/bin/x86_64-w64-mingw32-libgfortran4-cxx11/x86_64-w64-mingw32-gcc --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -std=gnu99  -lws2_32 -lmsvcr100 -shared -o gmt.dll -Wl,--out-implib,gmt.dll.a -Wl,--major-image-version,6,--minor-image-version,2 -Wl,--whole-archive CMakeFiles/gmtlib.dir/objects.a -Wl,--no-whole-archive @CMakeFiles/gmtlib.dir/linklibs.rsp

which does not work.

[joa-quim]

Could you give it another try? I fixed the gethostname and the _set_fmode issues upstream where I was finally able to build GMT with MinGW. But note that those changes are in GMT#master only

[giordano]

There are still lots of mistakes in the source code of GMT: all occurrences of WIN32 should be either _WIN32 if they apply to any Windows target, or _MSC_VER if it's only for the Microsoft Visual Studio Code compiler. You shouldn't even expect compilers to wrongly define macros not prefixed with a leading underscore, that's forbidden by the C standard, but non-standard compliant compilers like MSVC might do what they want. Besides, WIN32 isn't even documented in https://learn.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=msvc-170

[joa-quim]

Hmm, let me see if I understand it. You revising(???) the GMT C code and not accepting it and therefore this attempt of creating a GMT_JLL cannot proceed while you do not approve GMT C, right?

We may all be very dumb on GMT (I'm not by far the largest contributor) but we are are a bit aware of the WIN32 vs _WIN32 mess. https://github.com/GenericMappingTools/gmt/blob/master/src/gmt_notposix.h#L148

And you may also do not like (or believe) to hear that along the GMT developing time we have found dozens of memory access bugs that were detected only by the non-standard compliant compilers like MSVC and that gcc and clang had happily let go.

[giordano]

No, you do not understand correctly. I am pointing out factual errors in the source code which prevent correct compilation. We cannot merge a PR if compilation fails for a platform (the alternative being excluding that platform altogether). And compilation for Windows was failing here :shrug:

[joa-quim]

And compilation for Windows was failing here

Yes, I'm aware of that but the point is: Is it still failing after the upstream fixes? And if yes, where/why.

[giordano]

No, I haven't tried, but I just looked for obvious mistakes.

[joa-quim]

but I just looked for obvious mistakes.

We could continue discussing this for ever, which I will not. At least while the tone maintains.

[boriskaus]

so windows compilation still fails as can be seen here with the following error:

[08:18:04] /workspace/srcdir/gmt/src/gmt_api.c:952:10: fatal error: TlHelp32.h: No such file or directory
--
  | [08:18:04]  #include <TlHelp32.h>
  | [08:18:04]           ^~~~~~~~~~~~
  | [08:18:04] compilation terminated.

Is that something that looks familiar? Interestingly also is that I do not get this error if I run the build script locally with:

julia build_tarballs.jl --debug --verbose x86_64-w64-mingw32

thanks!

[joa-quim]

Nope, first time I see it. I do find that header file in

C:\msys64\mingw64\include\tlhelp32.h

but in lower case. Case should not matter on Windows (I think). I also see that gcc7 is being used. Can it be that this version of MSYS is too old?

[giordano]

MinGW consistently uses lowercase names everywhere, for both header files and libraries. That's the point of the patch added to this PR (which I had recommended upstreaming, but I guess that hasn't happened).

[joa-quim]

OK, #include <tlhelp32.h> now it is.

[boriskaus]

thanks. The patch file suggests that we should also change <Windows.h> to <windows.h> at a couple of places. See this patch

[joa-quim]

OK, a PR to fix that Windows.h vs windows.h is on its way.

Done.

[joa-quim]

Fine, things seam to be advancing (thanks a lot) but when it works there are a couple things more that need to be tuned. First e want also this cmake setting set (GMT_ENABLE_OPENMP TRUE) and then there are the coastlines and country polygons cases. These are separate from the GMT repo and what happens is if they aren't found on the first use GMT downloads them. But this is not a very pleasant surprise as it will take some time to do it. The best is to take case of this at the building/installing step

Have a look at the lines around https://github.com/GenericMappingTools/gmt/blob/master/ci/build-gmt.sh#L45 which what is done for the CI machinery.

Then I think these are automatically set

set (COPY_GSHHG TRUE)
set (COPY_DCW TRUE)

but I never know.

[joa-quim]

We need to add the Ghostscript_jll to the list of de dependencies.

Dependency(PackageSpec(name="Ghostscript_jll", uuid="61579ee1-b43e-5ca0-a5da-69d92c66a64b"))

[boriskaus]

Do you need the executable or the dynamic library? Same with gmt: should the dynamic library be shipped as well?

[vchuravy]

Generally for a jll we will want the dynamic library if not now then eventually.

[joa-quim]

Do you need the executable or the dynamic library? Same with gmt: should the dynamic library be shipped as well?

We need all three dynamic libraries that are produced: libgmt, libpostscriptlight and the supplements. This last one is a plugin library and the normal install step places it under lib/gmt/plugins on *nix and under bin\gmt_plugins on Windows.

The gmt and docs executables are tinny ones that are handy to have too.

[boriskaus]

ok, this finally compiles now.

Few points:

1. I did not find a binary called doc, so that is not added
2. supplements is a *.so file on linux/mac and a *.dll on Windows. I now copied this to /bin/ without the extension as it did not seem to work with the extension. Not sure whether this will be an issue with GMT.jl
3. We now download dcw and gsshg as external dependency. I don't know whether (and where) this is later downloaded when GMT_jll is installed. Perhaps @giordano can comment on this?

[giordano]

supplements is a *.so file on linux/mac and a *.dll on Windows.

What are those files? Shared libraries? Plugins? Else?

I now copied this to /bin/ without the extension as it did not seem to work with the extension. Not sure whether this will be an issue with GMT.jl

Can you elaborate what "did not seem to work" means?

[joa-quim]

As I said above (probably was writing at same time as you guys) supplements is a plugin and if its placed where I said then the libgmt knows where to find it, otherwise one must create a config file and set a variable pointing to where it lives. Much better to have in its default location.

The two examples in https://www.generic-mapping-tools.org/GMTjl_doc/geophysics/ are created by the code inside supplements.

[joa-quim]

I did not find a binary called doc, so that is not added

Sorry, its docs not doc. It's used only to make access to documentation (GMT, not GMT.jl) easier from command line.

[joa-quim]

I have this for my local WSL build. I don't see one in this build recipe. Maybe it facilitates (that is, avoids it) the moving around files. -DCMAKE_INSTALL_PREFIX=~/programs