User-defined metafunctions based on dynamic library loading

[DyXel]

trafficstars

This is a follow-up of #797, #907, #909 and all other related issues/discussions regarding metafunctions I might have missed. It is not necessary to have all of the previous context to follow this PR, although it would help a lot.

Special thanks to @JohelEGP for the original idea and implementation (see references above), @MaxSagebaum for #809 which is included in this PR and @edo9300 for helping with the DLL loading implementation.

Introduction

A metafunction is a special kind of function which is invoked on a declaration's reflection, and participates in defining its meaning (more in the documentation). This PR implements a mechanism that allows users to define and apply these kind of functions, as opposed to being only writable within cppfront's code.

The basic idea behind this design is that the definition of a metafunction can be compiled separately to a dynamic library (.so on Unix-like, .dll on Windows), targeting the reflection API, which can then be loaded and used by cppfront when processing further code. The major advantage of doing it this way is that metafunctions are regular code, and thus are not limited to the usual C++ compile-time evaluation restrictions, this makes it possible to use third-party libraries or execute any arbitrary code (e.g.: Open and write files, call a remote service, etc.).

Of course there are caveats as well, one in particular is that you can't define a metafunction and use it in the same compilation step (e.g.: In the same file), as the code would have needed to be compiled by the C++ compiler before cppfront can load it.

Basic Usage

Here is an example with GCC as base C++ compiler, build systems can automate the tedious parts:

metafunctions.cpp2:

greeter: @meta (inout t: cpp2::meta::type_declaration) = {
  t.add_member($R"(say_hi: () = std::cout << "Hello, world!\nFrom (t.name())$\n";)");
}

main.cpp2

my_class: @::greeter type = { }
main: ()                  = my_class().say_hi();

Steps

1. Compile cppfront with -rdynamic: g++ cppfront.cpp -std=c++20 -rdynamic -o cppfront

2. Transpile metafunctions.cpp2: ./cppfront metafunctions.cpp2

3. Compile metafunctions.cpp to a dynamic library named libmetafunctions.so:

g++ -std=c++20 -I../include/ -shared -fPIC -o libmetafunctions.so metafunctions.cpp

6. Transpile main.cpp2: CPPFRONT_METAFUNCTION_LIBRARIES=./libmetafunctions.so ./cppfront main.cpp2

7. Compile main.cpp: g++ -std=c++20 -I../include/ main.cpp

8. Run the resulting binary (./a.out), which outputs:

Hello, world!
From my_class

Implementation Details

Features

The main ingredients are a set of orthogonal features that combine to provide the full application:

Reflection API to generate non-member declarations

append_declaration_to_translation_unit is added, which allows a metafunction to generate code outside of the declaration that it is being applied to. Useful for things like factory functions and more.

Support applying metafunctions to functions and the @meta metafunction

The ability to apply a metafunction to a function declaration is enabled, and a @meta metafunction is added to mark would-be metafunctions in order to perform automatic registration.

Reflection API to obtain a declaration's fully qualified name

fully_qualified_name() is added, which allows obtaining the unique and final identifier of any declaration node, which can be used to refer to a specific entity from any place in code.

Public reflection API header and the internal foreign interface metafunction

The declaration of the reflection API is entirely moved to a public place that can be used by everybody including the compiler itself, it can be thought of as a "synopsis", but it is compilable code.

An internal compiler metafunction is added @_internal_foreign_interface_pseudovalue (obnoxious name on purpose), which automatically wraps the aforementioned declarations of the public header into something that allows for the full definition of the API to live somewhere else outside the library (i.e.: Inside cppfront).

The compiler's internal reflection code is refactored to provide the implementation of the interfaces created by the processing of the public header, allowing complete decoupling while keeping value semantics for the API, which also means not having to modify existing metafunctions.

Dynamic library loading and look-up mechanism

A standalone header (source/dll.h) is added that permits loading dynamic libraries on demand. Additional code that ties the loading of the dynamic libraries with the registration of metafunctions, as well as looking up names when attempting to apply a metafunction is added.

Putting All Together

When @meta is applied, it does a few things:

• Add the support header cpp2reflect_api.h.
• Generates an anonymous entity which receives the fully qualified name of the function as a string and its address.

The generated entity is of type register_metafunction, which is declared within cpp2reflect_api.h, and its constructor will be called when the library is loaded. The implementation of the overloads for this constructor are provided within cppfront.

Before loading a library, cppfront sets itself up so when the implementations of register_metafunction are called, they dispatch to a internal object which does bookkeeping and ensures the metafunctions can be found and called when needed.

Once cppfront tries to apply a metafunction, it will create the internal context (compiler_services_base) and a reflection API object, by filling in the interfaces with its internal implementation (provided by the renamed _impl classes in reflect.h2). In essence, the object is made up of "vtables", and the function pointers are all provided by cppfront at runtime. Prepped with this, it can proceed to do name look-up as before, it first attempts to use the internal metafunctions (the ones with nice name), followed by looking up in the loaded libraries.