buildsys: should we drop libtool and simply use our own list of build rules?

[fingolfin]

The main reason to use libtool for us is that it provides a uniform way to generate shared libraries and loadable modules (for kernel extensions). That used to be quite complicated because every vendor had their own way of doing it, and they differed, sometimes by a lot.

But these days, things are much simpler: essentially, you have to cover Linux, macOS, Windows, *BSD.

And using libtool has its costs, too:

• create .la and .lo files and .libs dirs
• involves a huge complicated shell script, intransparent
• relatively slow (doesn't matter much for us)
• complicates installation of GAP, as gac currently calls into libtool, so we'd have to install a copy of it (and some distros frown upon that, or at least it used to be that way)

I've been using this as a replacement in GAP.jl for some time for building loadable modules:

        if Sys.islinux() || Sys.isfreebsd()
            c_compiler = "$(CC) -fPIC -DPIC"
            cxx_compiler = "$(CXX) -fPIC -DPIC"
            c_dyn_linker = "$(CC) -shared -fPIC -DPIC"
        elseif Sys.isapple()
            c_compiler = "$(CC) -fno-common -DPIC"
            cxx_compiler = "$(CXX) -fno-common -DPIC"
            c_dyn_linker = "$(CC) -bundle"
        else
            error("OS not supported")
        end

Note that it lacks code for Windows, but I am sure we could overcome that. It also doesn't cover building the libgap.so shared library, but this requires rather similar code.

The main downside to dropping libtool is that we may loose support for some operating systems which someone (who??) out there might still care about. But I don't expect many (if any) of these, and adding back support for such a system generally is not much harder than the above.

Note that this would essentially return us to how we did things before the new build system was introduced in GAP 4.9, so it wouldn't be really something new either. Specifically, we had this (I've removed a bunch of useless parts dealing with NeXTStep, Irix etc.):

AC_DEFUN(GP_PROG_CC_DYNFLAGS,
[AC_CACHE_CHECK(dynamic module compile options, gp_cv_prog_cc_cdynoptions,
 [ case "$host-$CC" in
    *cygwin* )
        gp_cv_prog_cc_cdynoptions="${ABI_CFLAGS}";;
    *-apple-darwin*gcc* )
        gp_cv_prog_cc_cdynoptions="-fPIC -Wall ${ABI_CFLAGS}";;
    *-gcc* | *-egcs )
        gp_cv_prog_cc_cdynoptions="-fPIC -Wall -O2 ${ABI_CFLAGS}";;
...
    * )    dnl ## if we don't recognise this compiler, guess some flags
        gp_cv_prog_cc_cdynoptions="-fPIC -O2 ${ABI_CFLAGS}";;
   esac 
 ])
...
 AC_CACHE_CHECK(dynamic module link flags, gp_cv_prog_cc_cdynlinking,
 [ case "$host-$CC" in
    *-apple-darwin*gcc* )
        gp_cv_prog_cc_cdynlinking='-g -bundle -bundle_loader ${gap_bin}/gap -lc -lm'" ${ABI_CFLAGS}";;
    *-apple-darwin*clang )
        gp_cv_prog_cc_cdynlinking='-bundle -g -bundle_loader ${gap_bin}/gap'" ${ABI_CFLAGS}";;
    *cygwin* )
        gp_cv_prog_cc_cdynlinking='-shared ${gap_bin}/gap.dll';;
    *-gcc )
        gp_cv_prog_cc_cdynlinking="-shared -g ${ABI_CFLAGS}";;
    *-clang )
        gp_cv_prog_cc_cdynlinking="-shared -g ${ABI_CFLAGS}";;
...
   esac 
 ])

[ChrisJefferson]

I would prefer to drop it -- we already have to do weird things on windows (copy DLLs built by packages into the .libs directory) to work around libtool issues I don't entirely understand.

[ChrisJefferson]

Just for reference, here is the weird code from semigroups's Makefile.am:

all-local: semigroups.la
        $(mkdir_p) $(top_srcdir)/$(BINARCHDIR)
if SYS_IS_CYGWIN
        cp .libs/semigroups.dll $(GAPINSTALLLIB)
if WITH_INCLUDED_LIBSEMIGROUPS
# Cygwin will only look in this directory for dlls
        cp libsemigroups/.libs/cygsemigroups-*.dll $(GAPROOT)/.libs
endif
else
        cp .libs/semigroups.so $(GAPINSTALLLIB)
endif

[wilfwilson]

No objections from me, but it's also not something I really know anything about.

[fingolfin]

I am currently looking into this. After a lot of back and forth, my plan right now is to keep GNU libtool for building libgap, but to use custom rules for building kernel extensions / gac. The latter is substantially easier than the former, and the only where we really benefit from dropping libtool (on the long run we might still want to get rid of GNU libtool, but there is no need to rush that)

[orlitzky]

Disclaimer: I know nothing about GAP's build system.

But I do have a lot of experience with packages that try to invoke libtool themselves, and usually, it's not worth all the trouble it causes. In an autotools build system, using libtool is automatic, and you get nice features like easy shared/static handling, library versioning, *_FLAGS support in automake, etc. But if you're building the libraries yourself you have to understand all of the crazy stuff libtool does.

(And yes, installing the shell script is annoying for distros because when it is generated, things like CC get hard-coded into the libtool script.)

One boon for this approach is that cygwin is becoming less relevant now that WSL is common. This leaves Windows, and a bunch of platforms that act more or less sanely.

[fingolfin]

I know nothing about GAP's build system.

Indeed :-). Please refer to the final section of README.buildsys.md.

[fingolfin]

Also note that this is not about "directly invoking libtool" (which we already do, quite successfully, it's not hard), but rather about not using it anymore (at least for some things).

Also, about 99% of the crazy stuff GNU libtool does these days is completely pointless and only makes compilation slower. Compatibility with broken SunOS, Ultrix, whatnot versions from the 90ies just isn't that appealing anymore these days...

Source: I've wrangled with GNU libtool (and autotools etc.) for 20+ years, have patches in there, etc. And while I usually defend them when people criticize them, that's only because they usually criticize them for the wrong reasons 😂 . Anyway: we only aim to support Linux, macOS, Windows (well, cygwin, though mingw would be a nice dream...), FreeBSD; anything beyond that is nice but purely optional and only interesting if there is anybody who actually needs it (note that my patches likely will retain support for them anyway, simply because we get it "for free" with gcc and clang... but we have no ways to test that, so that's that)

[orlitzky]

I know nothing about GAP's build system.

Indeed :-). Please refer to the final section of README.buildsys.md.

In retrospect, my comment is not very clear. I'm not suggesting that you autotool-ize everything (which isn't possible with gac). The "this approach" I was referring to is yours, namely that you stop using libtool for some things.

Also note that this is not about "directly invoking libtool" (which we already do, quite successfully, it's not hard), but rather about not using it anymore (at least for some things).

Invoking it during the build isn't hard, but as you know it causes problems post-install. We've been able to address some of these issues by using a system-wide slibtool, but GAP can't rely on that. The GNU libtool script that is generated, on the other hand, is married to the GAP build, so it isn't 100% appropriate to use in gac. So specifically in gac I think eliminating libtool would be a win.

[fingolfin]

Ok then it seems we fully agree :-)

[fingolfin]

As an update for this, gac stopped using libool in GAP 4.12.0. We still use it to build libgap, though. I am working on replacing it there, too.