feat: evaluate program-defined metafunctions (based on #797)

[JohelEGP]

trafficstars

feat: evaluate program-defined metafunctions (based on #797)

A metafunction is normal Cpp2 code compiled as part of a library. When parsing a declaration that @-uses the metafunction, the library is loaded and the metafunction invoked on the declaration.

The reflection API is available by default to Cpp2 code (via cpp2util.h). The implementation of the API is provided by the cppfront executable. For this to work, compiling cppfront should export its symbols (for an explanation, see https://cmake.org/cmake/help/latest/prop_tgt/ENABLE_EXPORTS.html).

For cppfront to emit program-defined metafunctions, the environment variable CPPFRONT_METAFUNCTION_LIBRARY should be set to the library's path.

For cppfront to load program-defined metafunctions, the environment variable CPPFRONT_METAFUNCTION_LIBRARIES should be set to the :-separated library paths of the used metafunctions.

Here is an example of program-defined metafunctions. The commands were cleaned up from the CMake buildsystem in #797.

metafunctions.cpp2:

greeter: (inout t: cpp2::meta::type_declaration) = {
  t.add_member($R"(say_hi: () = std::cout << "Hello, world!\nFrom (t.name())$\n";)");
}

main.cpp2:

my_class: @greeter type = { }
main: ()                = my_class().say_hi();

Build cppfront:

g++ -std=c++20 cppfront.cpp -o cppfront
    # Note: check that we don't need to specify these flags explicitly, if they're defaults
    # g++ -std=c++20 -o cppfront.cpp.o -c cppfront.cpp
    # g++ -Wl,--export-dynamic -rdynamic cppfront.cpp.o -o cppfront

Build metafunctions:

CPPFRONT_METAFUNCTION_LIBRARY=libmetafunctions.so ./cppfront metafunctions.cpp2
g++ -std=c++20 -fPIC -o metafunctions.cpp.o -c metafunctions.cpp
g++ -fPIC -shared -Wl,-soname,libmetafunctions.so -o libmetafunctions.so metafunctions.cpp.o

Build and run main:

CPPFRONT_METAFUNCTION_LIBRARIES=libmetafunctions.so ./cppfront main.cpp2
g++ -std=c++20 main.cpp -o main
./main

Output:

metafunctions.cpp2... ok (all Cpp2, passes safety checks)

main.cpp2... ok (all Cpp2, passes safety checks)

Hello, world!
From my_class

[JohelEGP]

The design paper will come later.

[JohelEGP]

I opened #909 for the design write-up.

[JohelEGP]

Now reflect.h2 doesn't have a dependency on parse.h. It is compiled as a pure Cpp2 header, taking advantage of https://github.com/hsutter/cppfront/issues/594#issuecomment-1793627053. reflect_impl.h2 has the remaining bits that depend on parse.h.

To regenerate reflect.h2, use

cppfront -p reflect.h2 -o cpp2reflect.h
mv cpp2reflect.h ../include/

I use std::any values to build the compilation firewall. The implementation, cpp2reflect.hpp, is #included at the end of reflect_impl.h2. This is why odr-uses of the reflection API requires linking to cppfront.

[DyXel]

If depending on Boost.DLL is undesirable, loading and using shared libraries is actually quite simple:

For POSIX:

• dlopen to open the SO
• dlclose to close the SO
• dlsym to get a symbol (function in this case)
• dlerror to get a error string for error checking

For Windows:

• LoadLibraryA to open the DLL
• FreeLibrary to close the DLL
• GetProcAddress to get a symbol (again, a function)
• A combination of GetLastError, FormatMessageA and LocalFree to get a error string for error checking

It doesn't get much more complicated than that if you just want to call C-named functions. Let me know if you'd like me to support on this.

[JohelEGP]

Let me know if you'd like me to support on this.

Thank you. I think we would all like this.

It would help to not have to depend on Boost.DLL if there is an implementation for the current platform.

For GCC, I can use my system's Boost.DLL, But for Clang, I have to build its dependencies from source due to ABI (https://github.com/hsutter/cppfront/discussions/797#discussioncomment-7492847).

[DyXel]

Alright, I'll write the changes and open PR against the branch in your repo, we can discuss further over there once its up 👍🏻

[JohelEGP]

The diff in QtCreator better shows how reflect.h2 changed into cpp2reflect.h2:

Diff

--- source/reflect.h2
+++ source/cpp2reflect.h2
@@ -15,7 +15,6 @@
 //  Reflection and meta
 //===========================================================================
 
-#include "parse.h"
 
 cpp2: namespace = {
 
@@ -33,58 +32,51 @@ compiler_services: @polymorphic_base @copyable type =
 {
     //  Common data members
     //
-    errors               : *std::vector<error_entry>;
-    errors_original_size : int;
-    generated_tokens     : *std::deque<token>;
-    parser               : cpp2::parser;
-    metafunction_name    : std::string = ();
-    metafunction_args    : std::vector<std::string> = ();
-    metafunctions_used   : bool = false;
+    data_: std::any;
+    private data: (this) -> forward _ = std::any_cast<std::add_lvalue_reference_t<const compiler_services_data>>(data_);
+    private data: (inout this) -> forward _ = std::any_cast<std::add_lvalue_reference_t<compiler_services_data>>(data_);
 
     //  Constructor
     //
     operator=: (
         out this,
-        errors_          : *std::vector<error_entry>,
-        generated_tokens_: *std::deque<token>
+        data_v: std::any
     )
     = {
-        errors = errors_;
-        errors_original_size = cpp2::unsafe_narrow<int>(std::ssize(errors*));
-        generated_tokens = generated_tokens_;
-        parser = errors*;
+        data_ = data_v;
+        assert<Type>( data_.type() == Typeid<compiler_services_data>(), "parameter 'data_v' must store a 'compiler_services_data'" );
     }
 
     //  Common API
     //
     set_metafunction_name: (inout this, name: std::string_view, args: std::vector<std::string>) = {
-        metafunction_name  = name;
-        metafunction_args  = args;
-        metafunctions_used = args.empty();
+        data().metafunction_name  = name;
+        data().metafunction_args  = args;
+        data().metafunctions_used = args.empty();
     }
 
-    get_metafunction_name: (this) -> std::string_view = metafunction_name;
+    get_metafunction_name: (this) -> std::string_view = data().metafunction_name;
 
     get_argument: (inout this, index: int) -> std::string = {
-        metafunctions_used = true;
-        if (0 <= index < metafunction_args.ssize()) {
-            return metafunction_args[index];
+        data().metafunctions_used = true;
+        if (0 <= index < data().metafunction_args.ssize()) {
+            return data().metafunction_args[index];
         }
         return "";
     }
 
     get_arguments: (inout this) -> std::vector<std::string> = {
-        metafunctions_used = true;
-        return metafunction_args;
+        data().metafunctions_used = true;
+        return data().metafunction_args;
     }
 
-    arguments_were_used: (this) -> bool = metafunctions_used;
+    arguments_were_used: (this) -> bool = data().metafunctions_used;
 
     protected parse_statement: (
         inout this,
         copy source: std::string_view
     )
-        -> (ret: std::unique_ptr<statement_node>)
+        -> _
     = {
         original_source := source;
 
@@ -116,7 +108,7 @@ compiler_services: @polymorphic_base @copyable type =
         //  Now lex this source fragment to generate
         //  a single grammar_map entry, whose .second
         //  is the vector of tokens
-        _ = generated_lexers.emplace_back( errors* );
+        _ = generated_lexers.emplace_back( data().errors* );
         tokens := generated_lexers.back()&;
         tokens*.lex( lines*, true );
 
@@ -124,20 +116,18 @@ compiler_services: @polymorphic_base @copyable type =
 
         //  Now parse this single declaration from
         //  the lexed tokens
-        ret = parser.parse_one_declaration(
+        ret := data().parser.parse_one_declaration(
                  tokens*.get_map().begin()*.second,
-                generated_tokens*
+                 data().generated_tokens*
                );
         if !ret {
             error( "parse failed - the source string is not a valid statement:\n(original_source)$");
         }
+        return ret;
     }
 
-    position: (virtual this)
-        -> source_position
-    = {
-        return ();
-    }
+    protected position: (this) std::any_cast<source_position>(vposition());
+    protected vposition: (virtual this) -> std::any = source_position();
 
     //  Error diagnosis and handling, integrated with compiler output
     //  Unlike a contract violation, .requires continues further processing
@@ -156,10 +146,10 @@ compiler_services: @polymorphic_base @copyable type =
     error: (this, msg: std::string_view)
     = {
         message := msg as std::string;
-        if !metafunction_name.empty() {
-            message = "while applying @(metafunction_name)$ - (message)$";
+        if !data().metafunction_name.empty() {
+            message = "while applying @(data().metafunction_name)$ - (message)$";
         }
-        _ = errors*.emplace_back( position(), message);
+        _ = data().errors*.emplace_back( position(), message);
     }
 
     //  Enable custom contracts on this object, integrated with compiler output
@@ -167,7 +157,7 @@ compiler_services: @polymorphic_base @copyable type =
     //
     report_violation: (this, msg) = {
         error(msg);
-        throw( std::runtime_error("  ==> programming bug found in metafunction @(metafunction_name)$ - contract violation - see previous errors") );
+        throw( std::runtime_error("  ==> programming bug found in metafunction @(data().metafunction_name)$ - contract violation - see previous errors") );
     }
 
     has_handler:(this) true;
@@ -206,7 +196,7 @@ type_id: @polymorphic_base @copyable type =
     template_args_count : (this) -> int         = n*.template_arguments().ssize();
     to_string           : (this) -> std::string = n*.to_string();
 
-    position: (override this) -> source_position = n*.position();
+    protected vposition: (override this) -> std::any = n*.position();
 }
 */
 
@@ -224,20 +214,36 @@ declaration_base: @polymorphic_base @copyable type =
 {
     this: compiler_services = ();
 
-    protected n: *declaration_node;
+    node_pointer: @copyable type =
+    {
+        n: std::any;
+
+        operator=: <T> (
+            implicit out this,
+            n_: T
+        )
+        = {
+            n = n_;
+            assert( n_, "a meta::declaration must point to a valid declaration_node, not null" );
+            static_assert( std::is_same_v<T, *declaration_node> );
+        }
+
+        operator*: (this) -> forward _ = std::any_cast<*declaration_node>(n)*;
+    }
+
+    protected n: node_pointer;
 
     protected operator=: (
         out this,
-        n_: *declaration_node,
+        n_: node_pointer,
         s : compiler_services
     )
     = {
         compiler_services = s;
         n = n_;
-        assert( n, "a meta::declaration must point to a valid declaration_node, not null" );
     }
 
-    position: (override this) -> source_position = n*.position();
+    protected vposition: (override this) -> std::any = n*.position();
 
     print: (this) -> std::string = n*.pretty_print_visualize(0);
 }
@@ -252,7 +258,7 @@ declaration: @polymorphic_base @copyable type =
 
     operator=: (
         out this,
-        n_: *declaration_node,
+        n_: declaration_base::node_pointer,
         s : compiler_services
     )
     = {
@@ -334,7 +340,7 @@ function_declaration: @copyable type =
 
     operator=: (
         out this,
-        n_: *declaration_node,
+        n_: declaration_base::node_pointer,
         s : compiler_services
     ) =
     {
@@ -349,7 +355,7 @@ function_declaration: @copyable type =
     has_move_parameter_named     : (this, s: std::string_view) -> bool = n*.has_move_parameter_named(s);
     first_parameter_name         : (this) -> std::string               = n*.first_parameter_name();
 
-    has_parameter_with_name_and_pass: (this, s: std::string_view, pass: passing_style) -> bool
+    has_parameter_with_name_and_pass: (this, s: std::string_view, pass) -> bool
                                                   = n*.has_parameter_with_name_and_pass(s, pass);
     is_function_with_this        : (this) -> bool = n*.is_function_with_this();
     is_virtual                   : (this) -> bool = n*.is_virtual_function();
@@ -421,7 +427,7 @@ object_declaration: @copyable type =
 
     operator=: (
         out this,
-        n_: *declaration_node,
+        n_: declaration_base::node_pointer,
         s : compiler_services
     ) =
     {
@@ -457,7 +463,7 @@ type_declaration: @copyable type =
 
     operator=: (
         out this,
-        n_: *declaration_node,
+        n_: declaration_base::node_pointer,
         s : compiler_services
     ) =
     {
@@ -592,7 +598,7 @@ alias_declaration: @copyable type =
 
     operator=: (
         out this,
-        n_: *declaration_node,
+        n_: declaration_base::node_pointer,
         s : compiler_services
     ) =
     {
@@ -1338,110 +1344,6 @@ print: (t: meta::type_declaration) =
 }
 
 
-//-----------------------------------------------------------------------
-//
-//  apply_metafunctions
-//
-apply_metafunctions: (
-    inout n     : declaration_node,
-    inout rtype : type_declaration,
-    error
-    )
-    -> bool
-= {
-    assert( n.is_type() );
-
-    //  Check for _names reserved for the metafunction implementation
-    for  rtype.get_members()
-    do   (m)
-    {
-        m.require( !m.name().starts_with("_") || m.name().ssize() > 1,
-                    "a type that applies a metafunction cannot have a body that declares a name that starts with '_' - those names are reserved for the metafunction implementation");
-    }
-
-    //  For each metafunction, apply it
-    for n.metafunctions
-    do  (meta)
-    {
-        //  Convert the name and any template arguments to strings
-        //  and record that in rtype
-        name := meta*.to_string();
-        name = name.substr(0, name.find('<'));
-
-        args: std::vector<std::string> = ();
-        for meta*.template_arguments()
-        do  (arg)
-            args.push_back( arg.to_string() );
-
-        rtype.set_metafunction_name( name, args );
-
-        //  Dispatch
-        //
-        if name == "interface" {
-            interface( rtype );
-        }
-        else if name == "polymorphic_base" {
-            polymorphic_base( rtype );
-        }
-        else if name == "ordered" {
-            ordered( rtype );
-        }
-        else if name == "weakly_ordered" {
-            weakly_ordered( rtype );
-        }
-        else if name == "partially_ordered" {
-            partially_ordered( rtype );
-        }
-        else if name == "copyable" {
-            copyable( rtype );
-        }
-        else if name == "basic_value" {
-            basic_value( rtype );
-        }
-        else if name == "value" {
-            value( rtype );
-        }
-        else if name == "weakly_ordered_value" {
-            weakly_ordered_value( rtype );
-        }
-        else if name == "partially_ordered_value" {
-            partially_ordered_value( rtype );
-        }
-        else if name == "struct" {
-            cpp2_struct( rtype );
-        }
-        else if name == "enum" {
-            cpp2_enum( rtype );
-        }
-        else if name == "flag_enum" {
-            flag_enum( rtype );
-        }
-        else if name == "union" {
-            cpp2_union( rtype );
-        }
-        else if name == "print" {
-            print( rtype );
-        }
-        else {
-            error( "unrecognized metafunction name: " + name );
-            error( "(temporary alpha limitation) currently the supported names are: interface, polymorphic_base, ordered, weakly_ordered, partially_ordered, copyable, basic_value, value, weakly_ordered_value, partially_ordered_value, struct, enum, flag_enum, union, print" );
-            return false;
-        }
-
-        if (
-            !args.empty()
-            && !rtype.arguments_were_used()
-            )
-        {
-            error( name + " did not use its template arguments - did you mean to write '" + name + " <" + args[0] + "> type' (with the spaces)?");
-            return false;
-        }
-    }
-
-    return true;
-}
-
-
 }
 
 }

[JohelEGP]

Thanks to the contribution of https://github.com/JohelEGP/cppfront/pull/1 by @DyXel and @edo9300 now we directly use the OS APIs for loading libraries. This means that Boost.DLL is no longer required, and a cppfront compiled as usual will have support for loading metafunctions. You still need to specify the libraries with metafunctions via an environment variable. I have updated the opening comment, which should be used as commit message when merging, to reflect this.

[JohelEGP]

There's an issue with regards to symbol visibility.

On Windows, symbols are not exported by default. For Visual Studio, we need to use one of four methods to export symbols (https://learn.microsoft.com/en-us/cpp/build/reference/export-exports-a-function?view=msvc-170).

In Cpp1, exporting a symbol usually entails decorating its declaration with a portable macro (here CPPFRONTAPI). Unfortunately, there's no way to specify that in the Cpp2 source reflect.h2. The other three methods for VS happen outside the source code, which means manually listing the declarations.

Unlike GCC and Clang, it seems like VS doesn't offer an option to export all symbols. So we need Cpp2 support to export library symbols, or to use one of the other three VS-specific methods, or, equivalently, a patch that applies the export macro to the generated declarations.

NB: It is compiling cppfront which should export the symbols as it compiles the definitions in cpp2reflect.hpp.

[JohelEGP]

So we need Cpp2 support to export library symbols,

I have another use for such a feature.

In my case, I'm using an implementation of @rule_of_zero (https://github.com/hsutter/cppfront/pull/808). Then I'm defining @sfml, which uses it (https://github.com/hsutter/cppfront/discussions/789#discussioncomment-7572313).

When compiling with the hidden visibility preset, loading the metafunction sfml fails. It works when compiling the module that declares sfml because it imports the module that exports rule_of_zero. But on load, the rule_of_zero in its body is diagnosed as an undefined symbol, as it isn't visible. Manually adding CPPFRONTAPI to the lowered declaration of rule_of_zero makes it work.

[JohelEGP]

I'm thinking of just adding a metafunction to lower a declaration with the CPPFRONTAPI macro.

There are similarities to export declarations of C++ modules (which will be an access-specifier in Cpp2, see https://github.com/hsutter/cppfront/issues/269#issuecomment-1464790572). I asked on the Cpplang Slack on how it relates to symbol visibility, but they are orthogonal features (see https://cpplang.slack.com/archives/C92GZLCSE/p1700658714626929).

Relatedly, we might eventually want to lower extern "C" declarations. #624 is also interested in using @c_api for something else.

Anyways, I'll try to think of some fitting name to get things moving here. Suggestions are welcome!

[DyXel]

When compiling with the hidden visibility preset, loading the metafunction sfml fails. It works when compiling the module that declares sfml because it imports the module that exports rule_of_zero. But on load, the rule_of_zero in its body is diagnosed as an undefined symbol, as it isn't visible. Manually adding CPPFRONTAPI to the lowered declaration of rule_of_zero makes it work.

I guess technically you could use the C declaration for this case, but you'd need to do the casting to void*, plus namespace handling would be out of the window. Or, since you can detect the signature of a metafunction already, metafunctions detected within a body of another could be lowered to the specific C-magic. Both very ugly hacks but could work for a POC.

Indeed there needs to be a way to mark stuff to use C-linkage and/or declare its symbol visibility (in general, a way of spelling this specific stuff, I am sure there are more out there), but I wonder, should that functionality be covered as you mention with a metafunction (c_api)? I didn't think of using them like that, feels like that is not what metafunctions are intended for, ~~and you are still modifying the signature of a function within cpp2 limits, no? It should error out saying its not valid code.~~

Edit: Saw the commit, of course adding a flag to modify the lowering behavior, that works! But I still am questioning whether metafunctions are the right tool for this.

[JohelEGP]

With commit 7107644ead0d837307d1cd5183119258e11c1938, by declaring rule_of_zero with @visible when using the hidden visibility preset, sfml loads successfully and everything works as when using the default visibility preset.

[JohelEGP]

Now I just need to overload if for types, use it in reflect.h2, and then VS will work (https://github.com/hsutter/cppfront/pull/907#issuecomment-1870757843).

[JohelEGP]

I guess technically you could use the C declaration for this case, but you'd need to do the casting to void*, plus namespace handling would be out of the window. Or, since you can detect the signature of a metafunction already, metafunctions detected within a body of another could be lowered to the specific C-magic. Both very ugly hacks but could work for a POC.

The undefined symbol being that of a metafunction is a coincidence. The general issue is that Cpp2, without @visible, can't be used to author a DLL (with proper hygiene, i.e., hidden visibility by default, like the Windows default). A metafunction couldn't use a name declared in another Cpp2 TU.

[JohelEGP]

Now I can use a cppfront compiled with the hidden visibility preset. With it, my uses of metafunctions in my project work just fine. So the Visual Studio issue should be solved now (https://github.com/hsutter/cppfront/pull/907#issuecomment-1870757843).

[JohelEGP]

The windows CI

• Warns (https://github.com/JohelEGP/cppfront/actions/runs/7351541625/job/20015062242#step:5:68):

  reflect.h2(81): warning C4251: 'cpp2::meta::compiler_services::data_': class 'std::any' needs to have dll-interface to be used by clients of class 'cpp2::meta::compiler_services' C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.37.32822\include\any(108): note: see declaration of 'std::any' reflect.h2(305): warning C4275: non dll-interface class 'cpp2::meta::declaration_base' used as base for dll-interface class 'cpp2::meta::declaration' reflect.h2(262): note: see declaration of 'cpp2::meta::declaration_base' reflect.h2(304): note: see declaration of 'cpp2::meta::declaration'

  The first, I can't solve. I suppose it isn't a problem.

• Errors (https://github.com/JohelEGP/cppfront/actions/runs/7351541625/job/20015062242#step:5:100):

  reflect.h2(693): error C2375: 'cpp2::meta::interface': redefinition; different linkage reflect.h2(693): note: see declaration of 'cpp2::meta::interface'

  And so on for the other declarations. I suppose I should also lower the macro to the non-defining declarations.

[JohelEGP]

I have a plan to solve the name lookup problem for good using only the current source file.

• Emit a metafunction symbol with its fully qualified name mangled.
• Lookup an @-used metafunction using the source order name lookup (since source_order_name_lookup is in to_cpp1.h, and sema.h's get_declaration_of isn't of help here, I'll have to repeat source_order_name_lookup during parse.h). If the lookup isn't locally unambiguous, I can diagnose why and how to fix it (there are many things to consider).
• Emit a static_assert to confirm that Cpp1 lookup finds the name we found. If you can @-use it, you should be able to use it (even if not strictly necessary by the implementation).

The only chance for surprise is when a user expects a non-local name to be found. In the rare case we have a local match, the generated code won't be what the user expects. That error shouldn't get past the static_assert. But it might just immediately break evaluating the next metafunction in a chain.

[JohelEGP]

I need the metafunction symbols exported by the libraries. I will fall back to using Boost.DLL to prototype the solution. It would be painful to use the C interfaces: https://stackoverflow.com/a/2694373.

Or maybe I should just go ahead and start emitting the extra semantic information (https://github.com/hsutter/cppfront/issues/909#issuecomment-1871286126).

[DyXel]

Or maybe I should just go ahead and start emitting the extra semantic information (#909 (comment)).

I think having a specific C function per TO/DLL that is able to tell whether or not it has the symbol, has value on its own, for example:

CPP2_C_API int cpp2_meta_library_has_metafunction(const char* name, size_t size) {
    static std::set<std::string_view> mfs = {"greeter", /*...*/};
    return mfs.count(std::string_view{name, size});
}

(...or alternatively, a function that gives you a list of strings from which you can build a look-up table)

Armed with a function like this you would be able to tell what was exported, but also, you could first check the existence of this same function before proceeding with anything else, granting the opportunity to give the user a good explanatory message, like "cpp2_meta_library_has_metafunction was not found in DLL 'x', are you sure 'x' is a cppfront meta library?".

Just my 2 cents though.

[JohelEGP]

Great idea, thank you!

[JohelEGP]

but I wonder, should that functionality be covered as you mention with a metafunction (c_api)? I didn't think of using them like that, feels like that is not what metafunctions are intended for

I don't disagree. But it's currently the most fitting place to specify properties for the declaration. I also want a @deleted instead of having to unsatisfactorily abuse a private overload (see the thread starting at https://github.com/hsutter/cppfront/issues/468#issuecomment-1627565647 and the referencing issues).

And potentially @all_freestanding, @freestanding, @freestanding_deleted, and @hosted (whose effect depend on __STDC_HOSTED__). Although those have usability limitations (it might not really be what we want):

• https://eel.is/c++draft/freestanding.item#3.sentence-2
• https://eel.is/c++draft/compliance#2.sentence-2
• https://eel.is/c++draft/compliance#3
• https://eel.is/c++draft/compliance#4.sentence-2

More generally, we still need a replacement for some uses of the preprocessor. Maybe Cpp1 reflection will help here.

[DyXel]

I don't disagree. But it's currently the most fitting place to specify properties for the declaration.

Yeah, for a POC is fine. I do want people trying this functionality to its maximum and give good feedback to Herb, but I do worry about its future, been thinking about this for a while now so I might as well share my opinion:

A user-defined metafunction right now can do way too much, after all, it is arbitrary Cpp1 code being compiled and executed. It was fine being only used internally by the cppfront compiler, but if we give users total freedom to do whatever they want within a metafunction (e.g. execute arbitrary code and generate side-effects) then it'll become a problem once cpp1 reflection/generation lands, assuming it would be a subset of what cppfront offers, there would be competition between what cppfront can do and what was standardized ("teach a man how to fish...").

Another thing that I don't like is the necessary double-pass introduced by this user-defined meta functionality, in order to use a user-defined metafunction you'd need to write Cpp2, which gets lowered to Cpp1, which then gets compiled, and then used somewhere else in Cpp2, this extends the usage from simple transpiler (cpp2 -> cpp1 -> compile and run!) to something more complicated (cpp2 -> cpp1 -> compile meta -> cpp2 -> cpp1 -> compile and run!), which might drive adoption away.

For the latter, I think in a perfect world, cppfront would be able to interpret and apply the metafunction itself without having to loopback, and then when cpp1 meta lands (hopefully a good implementation with feedback received from this experiment!), we start generating cpp1 code instead of interpreting, and so users would be minimally affected.

[JohelEGP]

then it'll become a problem once cpp1 reflection/generation lands, assuming it would be a subset of what cppfront offers, there would be competition between what cppfront can do and what was standardized

For C++26 (P2996), the feature sets would be disjoint. In the future, we should have metafunctions (meta classes?) in Cpp1. That should subsume Cpp2 metafunctions, and we should migrate to using that Cpp1 feature.

[JohelEGP]

Or maybe I should just go ahead and start emitting the extra semantic information (#909 (comment)).

I think having a specific C function per TO/DLL that is able to tell whether or not it has the symbol, has value on its own, for example:

CPP2_C_API int cpp2_meta_library_has_metafunction(const char* name, size_t size)

Now I need this function to have a unique name per Cpp2 source with metafunctions.

A library can be composed of multiple source files. So I can't use the library path for the unique name. When loading the library, I can't map back to its source files.

I could use the absolute path of the Cppx source file. But without the abstraction to loop over a DLL's symbols, on top of needing the library path when loading a metafunction, I would also need the Cppx source files that make it up. The source of this information would be the caller of cppfront.

Emitting extra semantic information would be cleaner and forward-looking (https://github.com/hsutter/cppfront/issues/909#issuecomment-1871286126). Of course, that still requires the caller of cppfront to give us a file and to forward that file to the compilation of dependent Cpp2 source files.

[JohelEGP]

• Emit a static_assert to confirm that Cpp1 lookup finds the name we found. If you can @-use it, you should be able to use it (even if not strictly necessary by the implementation). -- https://github.com/hsutter/cppfront/pull/907#issuecomment-1871675196

For this sanity check to be viable in a non-modules world, metafunctions need to be authored in a pure Cpp2 .h2 header. The implementing .hpp header needs to be #included in a library's source file. A metafunction's @-use requires its .h2 header to be #included and its implementing library to be linked to. See https://github.com/hsutter/cppfront/issues/594#issuecomment-1793627053 for details on this .h2 header usage. I would also need to emit the loadable symbol in Phase 2 "Cpp2 type definitions and function declarations".

[JohelEGP]

With regards to https://github.com/hsutter/cppfront/pull/907#issuecomment-1872644205. There's another way to support multi-source libraries/C++ modules with TMP. The function that returns the list of symbols would be named the same on all sources but be a template dependent on a compile-time counter. The source compiled with CPPFRONT_METAFUNCTION_LIBRARY (the Cpp2 source of the library which includes the implementing .hpp headers, and the module interface unit for a C++ module) would have the uniquely-named symbol that returns the aggregated list.

[JohelEGP]

@hsutter This is ready for review.

[JohelEGP]

This is what might be able to remove the need for CPPFRONT_METAFUNCTION_LIBRARY (https://github.com/hsutter/cppfront/issues/909#issuecomment-1885005866):

IIUC, that inverts the logic so that plugins register themselves, right?

Yes, mostly. The application still needs to know that libraries to load but this process is just reduced to system calls to load the library and find one "C" function with a known name.

[DyXel]

This is what might be able to remove the need for CPPFRONT_METAFUNCTION_LIBRARY (#909 (comment)):

IIUC, that inverts the logic so that plugins register themselves, right?

Yes, mostly. The application still needs to know that libraries to load but this process is just reduced to system calls to load the library and find one "C" function with a known name.

https://github.com/hsutter/cppfront/pull/907#issuecomment-1871737914 I briefly mentioned something similar here.

Thinking about it, what you'd need is a static object for which you can register the metafunctions automatically when the DLL is loaded, then you can have a per-DLL function with that unique name that gives you back the mapping between a name and the actual metafunction. it would also be a good place (needed even?) for a "teardown", as we discussed.