Abandon dynamic linking

[mgree]

It has been tremendously painful working with ctypes: it's been hard to get things to work locally, nevermind in CI. Each OPAM version bump burns a day debugging linking issues.

The solution seems clear to me: abandon dynamic linking.

Chapter 20: Interfacing C with OCaml is the general resource we want.

We should simply write accessor functions for getting each field out of each struct. Each struct is an opaque type. Annoying, but straightforward enough.

It's tempting to use an Abstract_tag, which tells the GC about the pointer, or Custom_tag, which lets us use GC hooks to finalize things, etc. But we're already manipulating the stackmarks appropriately, and we don't want the OCaml GC mucking about in there. So: just return the raw pointer.

[yallop]

Out of interest, have you considered using ctypes without dynamic linking?

[mgree]

No, I hadn't, because I didn't know you could. You've got my attention! How does it work?

[yallop]

It works like this:

1. you use the same ctypes API to describe structs, functions, etc.
   (so you should be able to keep your existing code mostly intact)
2. you place the descriptions from step 1 inside functors that accept modules of type Ctypes.TYPE (for the structs) and Ctypes.FOREIGN (for the functions).
3. you pass the functors to Cstubs.write_ml and Cstubs.write_c to generate code that you can statically link into your program.

Conceptually, the idea is to give the ctypes bindings DSL a different interpretation so that it works as a compiler than an interpreter. The tradeoffs are similar to the interpreter/compiler tradeoff, too: it's a bit more work to set up the build, but in return you get faster code and better type-checking (since the generated code is checked against library headers).

If you decide to try this route there are various examples around, such as ocaml-mariadb or ocaml-yaml that it might be useful to copy.

[mgree]

That sounds very promising! I'll take a look.

I would need ocamlmklib call that just included the .a file from the original library along with the .o from those generated stubs, right? I'd generate the stubs at build time and then be able to ship a single, statically linked .cmxa?

[yallop]

I would need ocamlmklib call that just included the .a file from the original library along with the .o from those generated stubs, right?

Yes, along with the .cmx code from the generated OCaml.

I'd generate the stubs at build time and then be able to ship a single, statically linked .cmxa?

Yes.

[mgree]

Resolved in #29.