When the ABI doesn't match the package version...

[minrk]

While investigating https://github.com/conda-forge/zeromq-feedstock/issues/36 I learned that the libsodium package version (e.g. 1.0.16) doesn't reflect the ABI version (24 in 1.-.15-16, 23 in 13, 18 in 1.0.8, etc.). In the meantime, I'm pinning run_exports with max_pin=x.x.x since I can't know the next patch version won't bump the ABI version, even though it probably won't.

So the question has been raised: how to pin abi compatibility in run_exports, when the version number of the package isn't enough.

A few options have been proposed on gitter:

• add a libsodium25 output, so the ABI version is in the package name (like debian)
• add a libsodium_abi output with the ABI version
• prepend the ABI major version in the library package version (libsodium-24.1.0.16)

[msarahan]

Interesting problem. Gfortran has this problem, too. If conda build needs some new functionality, we can do that.

[minrk]

The simplest version in my mind would be to have the ABI version (just int for major version? Or full version string?) as additional metadata in the package and automatically add it to run_exports, if defined. I'm not quite sure how to define such a field or how the conda solver could take it into account.

libtool has a unique three-field versioning system:

• current
• revision
• age

These numbers have some unusual if logical definitions, but the semver way to write a given libtool version would generally be current-age.age.revision, where a package built against x.y is runtime-compatible with newer builds (not necessarily older) as long as current - age remains constant: [x.y,x+1.0) (revision is irrelevant to compatibility). I think encoding the full ABI compatibility info is probably overkill and the issue ought to be covered by adding just the major ABI version to the existing pin_compatible logic on package version. This should only really be required by libraries where the ABI version can change on patch releases, because the existing pin_compatible logic suffices as long as a given package triggers at least a minor revision when the bump the ABI.

This all assumes that conda package managers know that they have to record the ABI version in meta.yaml and actually do it (I wouldn't have for libsodium until the above zeromq issue, and I'll probably forget to update).

What would be double-awesome if is conda-build could automatically detect the ABI version (e.g. by the package stating that it installs "libsodium.dylib" and conda-build looks at the installed files and sees that libsodium.dylib is a symlink to libsodium.24.dylib (or libsodium.so.24) and automatically record the ABI version as "24" in package metadata and appropriately in run_exports. That way, package maintainers wouldn't need to manually verify the ABI version for each package update, which I'm a bit skeptical would really happen.

So if I had to only add to my meta.yaml:

build:
  abi_library: "libsodium"
  # or ${PREFIX}/lib/libsodium${SHLIB_EXT} if meta.yaml should specify the full path

and conda-build produced an appropriate abi-pinning metadata and run_exports, that would be awesome.

Here's a sketch of the abi-detection-by-symlink:

import os
import re
import sys


macos_abi_pat = re.compile("lib[^\.]+\.([\d\.]+)\.dylib$", re.IGNORECASE)
linux_abi_pat = re.compile("lib[^\.]+\.so\.([\d\.]+)$", re.IGNORECASE)

abi_pat = macos_abi_pat if sys.platform == 'darwin' else linux_abi_pat
SHLIB_EXT = '.dylib' if sys.platform == 'darwin' else '.so'


def detect_abi_version(libname):
    """Return the ABI version of a library
    
    given 'libzmq' return '5' as its ABI version
    because $PREFIX/lib/libzmq.dylib is a symlink to libzmq.5.dylib
    or $PREFIX/lib/libzmq.so is a symlink to libzmq.so.5
    """
    lib = os.path.join(sys.prefix, 'lib', libname + SHLIB_EXT)
    if not os.path.exists(lib):
        raise OSError("%s does not exist" % lib)
    if not os.path.islink(lib):
        print("%s is not a symlink, no ABI info found" % lib)
        # libfoo.dylib is not a symlink, no ABI version to find
        return
    target = os.readlink(lib)
    target_libname = os.path.basename(target)
    match = abi_pat.match(target_libname)
    if not match:
        print("No ABI found in %s" % target_libname)
        return
    return match.group(1)

(demo notebook)

[jakirkham]

This can get a bit tricky. For instance, see this comment about OpenBLAS.

That said, it would be nice to have some automated detection like what you have described here.

[djsutherland]

Rather than symlinks, one could check the actual SONAME. Something like this: https://gist.github.com/lebedov/5111696

[carterbox]

I drafted a CFEP to propose putting the ABI major version in the build string. I'd appreciate thoughts on the proposal or any further ideas on this topic.