Bootstrapping `lib.ml` via the foreign function interface

[dhil]

Lately, I have been working on making the internal handling of names hygienic in the compiler. I have coupled this work with the introduction of compilation units (#603). At this point, I am close to having something mergable. Most of the problems with names stems from the fact that you cannot reliably shadow names originating from lib.ml in Links as those names inadvertently have a super special status.

I have been experimenting with different ways of eliminating the need for lib.ml altogether. I have finally converged on a design that will allow me to treat lib.ml as any other user-definable compilation unit or module. Here I would like to outline my design and invite discussion.

I would like to declare all of the primitive functions in a Links source file using the foreign function interface. To make this clean, I need to modify the foreign function interface slightly. The foreign function interface has some problems:

• Every foreign function is assumed to live on the client side. This is unnecessary limiting.
• The source name of the binder is assumed to be the object name. As a consequence, only one foreign function can be named foo in the whole program. This is anti-modular.
• The filepath to the object file (implementation) is hard-coded into the Links source program. This path often depends on which directory Links is executed from. This is anti-modular.

I would like foreign functions to be able to live on the server side in order to declare functions like writeToSocket. I want the user to explicitly state the name of the function in the object language. Finally, I want object files to be explicitly given as commandline options.

Let me exemplify this. Here is a user program which declares one foreign function foo

# foo.links
alien javascript foo client : () -> () = "$bar";

foo() # calls $bar on the client.

The user has declared the function foo, whose JavaScript source name is $bar. Moreover, the function foo is a client side function, so if it is invoked on the server, then a remote call will be generated. To run foo.links, the user must link the appropriate JavaScript implementation for $bar. Suppose the implementation file is named "bar.js":

$ ./links --endpoint "/js" --jslib path/to/bar.js foo.links

Here we register /js as an endpoint on the Links webserver instance such that when /js/bar.js is requested it will serve path/to/bar.js. This is essentially what serveStatic does. The main difference is that serveStatic modifies the global state as the program runs, which is anti modular. The commandline approach modifies the global state once, and only when the whole program has been assembled.

The --endpoint declarations on the commandline are intended to be lexically scoped such that

$ ./links --endpoint "/js" --jslib path/to/bar.js --endpoint "/js2" --jslib baz.js --jslib /tmp/quux.js foo.links

resolves the following requests as

• /js/bar.js => path/to/bar.js
• /js2/baz.js => baz.js
• /js2/quux.js => /tmp/quux.js
• /js/baz.js => 404 Not Found

If no --endpoint declaration precedes an --jslib then the default endpoint / is assumed.

To declare all of lib.ml using the foreign function interface, I would like to be able to say that something is primitive. For example, the following is a subset of lib.ml as I envisaged it being declared in a Links source file

# lib.links
alien primitive {
  writeToSocket server : (String, Socket) ~> () = "%writeToSocket";
  dump client : (a) ~> () = "_LINKS.dump";
  == : (a, a) -> Bool = "%links_eq";
}

Here we declare writeToSocket as a primitive server-side only function. That is calling it from the client will generate a remote call. Conversely, dump is a client-side only function. The polymorphic equality function == has implementations on both the server and client sides. This is sufficient to bootstrap lib.ml in the frontend of the compiler. For now, the backend would still need some special knowledge, because the implementations of alien primitive functions must be known. The user should not/cannot provide implementation files for things declared as alien primitive. For the next iteration of the FFI, I would like to be able to link against their implementations (in C, OCaml, etc) such that they need not be defined internally in the compiler.

[dhil]

(Originally, I wanted to use native rather than primitive hence why I investigated #784.)

[dhil]

Here is lib.ml as a Links source file: https://github.com/dhil/links/blob/bootstrap-lib/lib/builtins.links