Shared Library Support

[trentforkert]

Pinging @mathstuf and @dcarp for comments/questions/concerns/better ideas on this. My apologies for the wall of text.

Currently, building D shared libraries with CMake is impossible, since when I started working on this, no D compilers could do shared libraries. Some of the flags and variables are in place, but will lead to errors if they are used.

However, progress has been made, with DMD >= 2.063 and LDC 0.14.0-alpha.1 (D 2.065) supporting building shared libraries on Linux. To that end, I've been looking into what would need to happen to get CMake building D shared libraries. One of my main goals when brainstorming this is that I want it to be future-proof (i.e., when building shared libs on other platforms starts working, CMake will already have the support infrastructure).

Some observations:

Different compilers/platforms/versions(!?) have different naming schemes:

• DMD on Linux, OS X and FreeBSD names its phobos library "phobos2", with appropriate platform prefix/suffix.
• DMD on 32-bit Windows has "phobos.a"
• DMD on 64-bit Windows has "phobos64.a"
• DMD moved 64-bit libs into lib64 in 2.065, but the naming stayed the same
• All DMD platforms/versions I've looked at put phobos and druntime into the same library, with no actual split in practice.
• GDC has a split phobos/druntime, naming them "gphobos2" and "gdruntime", with appropriate platform prefix/suffix
• LDC >= 0.14.0-alpha.1 has split phobos/druntime
• LDC >= 0.14.0-alpha.1 calls phobos "phobos2-ldc", and druntime "druntime-ldc", with appropriate platform prefix/suffix
• LDC < 0.14.0-alpha.1 also had split phobos/druntime libs, but the phobos static library bundled druntime without needing the druntime static lib.
• LDC < 0.14.0-alpha.1 calls phobos "phobos-ldc", and druntime "druntime-ldc", with appropriate platform prefix/suffix

Miscellaneous observations:

• LDC doesn't (currently?) support -fPIC, but will accept -L-fPIC
• Linux SO version suffixes are inconsistent between DMD and LDC. DMD uses .0.65 and 0.65.0 where LDC uses .65 and .2.0.65. Frankly, I don't trust either of these schemes to stay consistent in the future.

With all that in mind (and/or out of the way), my plan is as follows:

• Add library detection for both phobos and druntime on all D compilers as part of compiler detection
  • With hints for "lib" and "lib32"/"lib64" above the compiler's directory as appropriate
  • Only look for expected names for a given compiler/platform
• Use -defaultlib= to disable the compiler's default library
  • On LDC, the flag is -nodefaultlib
  • On GDC, it is -nophoboslib
• Explicitly link the found phobos/druntime
• If CMake doesn't find druntime, assume it is bundled into phobos
• If CMake doesn't find phobos, there are some options: (I'm currently leaning toward option 3)
  • Option 1: Assume that is fine
  • Option 2: Give a warning, but try to continue anyway
  • Option 3: Give an error
  • Option 4: Make phobos optional (May be desirable, but I can't find an equivalent for libc or libstdcxx)
• Implement miscellaneous fixes to do things like avoid defines (-Dfoo_EXPORTS) being passed and fix sonames, and possibly other problems I don't know about yet

This approach gives the following benefits:

• Shared phobos/druntime will be used if they exist, even if the compiler would default to static libs (like dmd does)
• Static libraries will be found and used otherwise, allowing things to work as they do currently
• With minimal things coded per-compiler or per-version, this should "just work" once D supports building shared libs on systems other than Linux
• (Mostly) transparent to the user. With a shared phobos/druntime, shared and static libraries will both work. With a static phobos/druntime, static libraries will work, but shared won't

And the following caveats:

• If multiple shared phobos versions are installed on Linux, this approach will take the system default, rather than the compiler version (if those are different), which may cause problems
• Depending on external libraries will likely require that they were built with the same compiler/version
• Related, mixing libraries that link shared phobos and libraries that link static phobos will lead to runtime problems (though, in my tests, things compiled fine)
• Making sure phobos/druntime are available to the end user is left to the developer/packager

[mathstuf]

Hmm. I've been building a shared library for the Android build of gunroar just fine with GDC. Haven't tested native libraries yet, but I at least get into the runtime in the emulator (where it subsequently crashes), but the library stuff seems to work fine.

[trentforkert]

Fresh build of GDC (upstream branch gdc-4.9) with the --enable-shared flag passed to configure allows for some insights:

• No shared phobos/druntime were built, which is what I expected
• Direct gdc -fPIC -shared foo.d fails
• Incremental gdc -fPIC -c foo.d && gdc -fPIC -shared -o libfoo.so foo.o fails
• Incremental gdc -fPIC -c foo.d && gdc -nophoboslib -fPIC -shared -o libfoo.so foo.o compiles, but fails at runtime

I think there must be something going on with Android here. According to the wiki[1] (which hasn't been updated in almost a year, but whatever), Phobos should be disabled, and druntime is incomplete. I suspect gdc sans libgphobos is behaving the same way as if you had passed the -nophoboslib flag, which would produce the results you are seeing in the emulator.

Add the fact that (unlike DMD and LDC) GDC ignores -Ddefines flags probably explains why your code compiles, and then crashes at runtime.

[1] http://wiki.dlang.org/GDC/Installation/Android#TODO

[mathstuf]

What crash do you see at runtime? I see a null deref in libc, but the back trace is garbled :/ .

[mathstuf]

You can try as well, but unfortunately getting an Android NDK with GDC is non-trivial, but the gunroar docs are a decent outline of my process for it. I'll check tomorrow if my published GDC branch is up-to-date.

[trentforkert]

Segfault. My test is fairly limited, though, just trying replicate what JNI would do using libdl in C. You need to initialize the D runtime before calling any of the shared D methods, but the needed rt_init hasn't been defined, so it segfaults.

Incidentally, it works if I build have a D main and link the library from there, but I suspect that would fail if my test was more than one function in one library.

[mathstuf]

I extern forward declare rt_init and gc_init and got further calling it before executing any D code. See src/android/jni/src/android_main.c in gunroar.

[trentforkert]

I've tried building gunroar for Android, and hit some snags:

• The only Derelict component whose cmake configs worked for me was DerelictUtil. The others couldn't get through configure, saying they couldn't find DerelictUtil, even when I passed -DDerelict_util_LIBRARY=/path/to/libDerelictUtil.a and -DDerelict_INCLUDE_DIR=../import to cmake. ../import/derelict contains symlinks to the various parts of Derelict in their individual repositories.
• Doing a manual compilation of DerelictSDL2 worked, but both DerelictGL3/DerelictGLES fail for me
  • GL3 has a static assert demanding I use GLES
  • GLES has various forward reference errors (EGLDisplay, EGLint, EGLDeviceEXT, and GLenum)

Since I couldn't get those built for Android, I haven't actually been able to try to build gunroar yet. I'll poke at it some more later.

I did notice, however, that I do at least have a libgphobos2.a in my NDK build, though I haven't been able to confirm whether or not it actually works.

For your particular use case (only one D shared lib), statically linking druntime/phobos might actually work once you have all the POSIX calls translated to Bionic's APIs.

[mathstuf]

The only Derelict component whose cmake configs worked for me was DerelictUtil. The others couldn't get through configure, saying they couldn't find DerelictUtil

ISTR a similar problem; not sure if I ever fixed it or just hacked the cache values…

GLES has various forward reference errors (EGLDisplay, EGLint, EGLDeviceEXT, and GLenum)

Hmm. I fixed these? It builds here with 98243b14540296f14e7bc8f94ccc48d527b39a14.

[trentforkert]

I've actually spent some time trying to implement this today, and I've hit a snag in the way I had planned on doing this, since I need variables that haven't been defined where I was trying to use them.

My new plan is to add a FindPhobos module (which will also find druntime). However, this means that any code which uses Phobos will need to also find it and link it manually.

I'd rather phobos/druntime remain implicit, but dmd's shared library support is an opt-in, with it defaulting to static phobos. So if you want shared libraries in D, and an executable in D, all using phobos, they all need to use the same phobos. Mixing static and shared phobos tends to result in runtime errors. This means that you have to tell dmd which phobos to use for every call. Using CMake's Find* functionality seems to be the only way I'll be able to get this to work.

Thoughts?

[mathstuf]

Look at COMPATIBLE_INTERFACE_BOOL. I'd set CMAKE_D_RUNTIME_{STATIC,SHARED} appropriately based on a target property for the user to set. The target usage requirements logic should take care of it from there.

[trentforkert]

I've fixed up the flags and things so that creating shared libraries works if the compiler supports it (AFAIK: only recent DMD or LDC on Linux x86{,_64}).

However, after encountering some more problems, I'm leaving any phobos detection stuff out, at least for now.

This requires, for example:

    export DC="/usr/bin/dmd -defaultlib=libphobos2.so"

to get DMD to work with shared libs.