modelica
Provide functions in ModelicaUtilities.h to external objects implemented as shared libraries
Currently it is not possible to access the functions in ModelicaUtilities.h from an ExternalObject that is implemented as a shared library in a generic way. Therefore I propose to introduce an ExternalObject
class ModelicaUtilityFunctions
extends ExternalObject;
function constructor
output ModelicaUtilityFunctions functions;
end constructor;
function destructor
input ModelicaUtilityFunctions functions;
end destructor;
end ModelicaUtilityFunctions;
into the MSL that is implemented by the tools and returns a pointer to the following structure.
#include "ModelicaUtilities.h"
typedef struct {
void (*ModelicaMessage)(const char *string);
void (*ModelicaFormatMessage)(const char *string, ...);
void (*ModelicaVFormatMessage)(const char *string, va_list);
void (*ModelicaError)(const char *string);
void (*ModelicaFormatError)(const char *string, ...);
void (*ModelicaVFormatError)(const char *string, va_list);
char* (*ModelicaAllocateString)(size_t len);
char* (*ModelicaAllocateStringWithErrorReturn)(size_t len);
// ...
} ModelicaUtilityFunctions_t;
This pointer can then be used by external objects to access these functions.
class MyExternalObject
extends ExternalObject;
function constructor
input ModelicaUtilityFunctions callbacks;
output MyExternalObject externalObject;
external"C" externalObject =
MyExternalObject_open(callbacks) annotation (
Library="MyExternalObject");
end constructor;
function destructor
input MyExternalObject externalObject;
external"C" MyExternalObject_close(externalArduino) annotation (
Library="MyExternalObject");
end destructor;
end MyExternalObject;
@t-sommer this is a good proposal to address a long-standing problem, see modelica/ModelicaSpecification#2191. We proposed an alternative solution there a few months ago, which just requires all Modelica tools and simulation runtimes to export the symbols of those functions, so they can be dynamically linked both at compile time and at run time. Unfortunately there was no feedback from tool vendors (other than OMC, that is) back then.
This already works in OpenModelica, we implemented and tested it in the ExternalMedia library. Unfortunately Dymola.exe does not export those symbols, so if some external functions are called during translation and they give an error, there is no way to properly report it. Hence, we conjured up a hack to pass those pointers to ExternalMedia static interface.
I'm not sure which of the two solutions is better. Our proposal requires to slighly change how you building the Modelica compiler and the simulation runtime, all tool vendors could do it right away. Yours is easier to understand for non-hackers like myself, but it requires an addition to ModelicaServices, so it could be released with 4.2.0 sometime next year.
For sure we should go ahead and implement one of these two ASAP.
BTW, this is yet another issue at the boundary between MAP-Lang and MAP-Lib.
@fedetftpolimi what do you think?
From a quick check it should allow both static and dynamic libraries to access the pointers to the modelicaUtilities functions, so from a C/C++ perspective it seems good.
However, the example is geared to external libraries based on external objects, for this approach to be complete is should also allow libraries based solely on external functions (such as ExternalMedia) to get the pointers.
This could be as simple as allowing to declare an external function taking the parameter input ModelicaUtilityFunctions callbacks;
This could be as simple as allowing to declare an external function taking the parameter input ModelicaUtilityFunctions callbacks;
Passing instances of external objects to functions is already allowed today, so this should not require any changes to the Modelica language.
@casella By the way, the two proposal are also not mutually exclusive. We may even support both allowing to retrieve the functions as symbols by the linker or pointers through the struct, even though there's no need to standardize both.
In any case, what we need is at least one solution implemented by all tool vendors asap (and one that isn't just "use static libraries" because that's a dead end when you account for dependencies).
For the sake of completeness:
- ModelicaUtilityFunctions_t of @t-sommer misses the functions for ModelicaDuplicateString and ModelicaDuplicateStringWithError.
- Mo-2191 holds more than the one proposal mentioned by @casella, e.g., by providing a specific shared object (named ModelicaExternalC) at run-time (which is implemented in SimulationX).
The proposal in https://github.com/modelica/ModelicaSpecification/issues/2191 we were referring to is the one @casella posted on on May 22, 2024: exporting symbols to allow automatic lookup by the dynamic linker/loader.
The idea of providing a tool-specific shared object as far as I know is not portable, and it forces C/C++ library developers to provide different versions of their library differing only by each version being linked with the shared object of a particular vendor, which does not scale well. E.g: the ExternalMedia build system does not provide versions of the library linked with the SimulationX shared object.
- ModelicaUtilityFunctions_t of @t-sommer misses the functions for ModelicaDuplicateString and ModelicaDuplicateStringWithError.
You're right, see Section 12.9.6.2 they were added in Modelica 3.5.
@t-sommer you may edit your original proposal to keep it consistent to the latest standard.
@HansOlsson the related issue modelica/ModelicaSpecification#2191 has been open for 7 years. I understand we now have two good proposals to fix it, so I would put it in the agenda of the MAP-Lang group. I have a slight preference for @t-sommers's proposal since it relies less on low-level C-code functionality (importing symbols in the external C functions) and uses explicitly defined Modelica functionality instead, but I'm not 100% sure it always works (see below).
Besides that, @t-sommer's proposal requires changes to the language spec (regarding the contents of ModelicaUtilities.h) and to the MSL (adding the ModelicaUtilityFunctions to ModelicaServices library), while mine only requires to change the language spec. But I don't see that as a big deal, with a modicum of coordination between the two groups 😃
Passing instances of external objects to functions is already allowed today, so this should not require any changes to the Modelica language.
@t-sommer maybe @fedetftpolimi has a point. It is not clear to me where the external object should be instantiated in some cases. Consider for example this MWE (which is relevant for ExternalMedia):
package MWE
partial function baseFunction
input Real x;
output Real y;
end baseFunction;
function function1
extends baseFunction;
algorithm
y := 2*x;
end function1;
function function2
extends baseFunction;
external;
end function2;
model M
replaceable function f = function1 constrainedby baseFunction;
Real y = f(time);
end M;
end MWE;
Now, we want the external function2() to be able to call ModelicaError. Where would we instantiate ModelicaUtilityFunctions so that we can pass the pointer to it to the function?
One idea could be to amend the specification of the external function interface by defining a pointer to an implicitly defined default instance modelicaUtilityFunctions of the ModelicaUtilityFunctions pre-defined external object, so you could write something like
function function2
extends baseFunction;
external "C"
y = extFunction2(x, modelicaUtilityFunctions.functions);
end function2;
double extFunction2(double x, void *callbacks)
{
if (x > 0)
return 2*x;
else
(ModelicaUtilityFunctions_t *)callbacks().ModelicaError("Negative input not allowed");
}
without the need of explicitly instantiating the modelicaUtilityFunctions external object. I'm not sure about the C syntax, but I guess the intent is clear, please correct it if necessary.
As I understand, that's the only way (besides exporting and importing symbols) to handle utility functions in purely function-based implementations, such as is the case of the setState_xx() functions in Modelica.Media. It would also be 100% backwards compatible with the existing external function interface.
Please try to avoid using void*.
To me the current C-calling interface for Modelica was designed to interface existing C-code, and then people wrote "small" C-glue functions (or what-ever you call them) to handle other data-types etc. The ModelicaUtilities header works for the glue-functions, but not when building larger libraries specifically for Modelica-libraries.
Obviously it is still possible to create glue-functions for a Modelica-library, but it becomes cumbersome to use it every time:
function function2
extends baseFunction;
external;
annotation(Library="...",, Include="
static void function2Print(const char*x) {
ModelicaFormatMessage(\"From function2:%s\", x);
}
extern double function2External(double x, void (*f)(const char*));
double function2(double x) {
return function2External(x, function2Print);
}"
end function2;
Oh, I looked more and ExternalMedia is written in C++ and uses ModelicaError.
That C/C++ interfacing is a real problem, that requires a different good solution which is completely different from what is discussed here.
I like @casella idea of a pre-defined instance of the ModelicaUtilityFunctions external object. We should avoid the situation of creating several redundant instances of the ModelicaUtilityFunctions object. Now that I think of it, this is the first time I came up with the need for singletons in Modelica, is there a clean way to implement them?
@HansOlsson true, whatever interface we come up with, we should avoid using C features that are incompatible with C++. So far a struct containing function pointers seems safe to me.
One can still dream and hope that someday a true direct interface between Modelica and C++ will be standardized, and in that case ModelicaUtilityFunctions would become a C++ class as well as a Modelica ExternalObject, but I'd rather have the struct asap, as the current mess for calling modelicaError and such makes it impossible to support all tools and that is a bigger problem.
I like @casella idea of a pre-defined instance of the ModelicaUtilityFunctions external object. We should avoid the situation of creating several redundant instances of the ModelicaUtilityFunctions object. Now that I think of it, this is the first time I came up with the need for singletons in Modelica, is there a clean way to implement them?
@HansOlsson true, whatever interface we come up with, we should avoid using C features that are incompatible with C++. So far a struct containing function pointers seems safe to me.
The problem is that calling ModelicaError isn't safe in C++. That is a real blocker.
Am I missing something? I always thought it was implemented by throwing an exception (C code can be compiled with exceptions on essentially all platforms) so that if there are C++ functions on the call stack the destructors will be called. I guess that since you say it's unsafe at least some tool uses setjmp/longjmp to implement ModelicaError, and that would indeed be bad. Can't we make it explicit in the standard that ModelicaError must be implemented using exceptions?
By the way, if this is not addressed, libraries such as ExternalMedia will always leak memory and likely crash the entire Modelica environment after attempting to recover from a ModelicaError...
Am I missing something?
Yes.
I always thought it was implemented by throwing an exception (C code can be compiled with exceptions on essentially all platforms) so that if there are C++ functions on the call stack the destructors will be called. I guess that since you say it's unsafe at least some tool uses setjmp/longjmp to implement ModelicaError, and that would indeed be bad. Can't we make it explicit in the standard that ModelicaError must be implemented using exceptions?
Exceptions are a C++-feature, we have standardized on a C-interface for simplicity (and many C++-compilers can disable exceptions for some reason) - so that even other languages like Java etc can be called.
The fact that the interface is in C does not forbid to implement ModelicaError by throwing an exception under the hood. If the exception propagates through C code it doesn't hurt, while if it propagates through C++ code it calls the proper destructors.
You can have the best of both worlds, you just chose not to. You can make a pure C interface and yet be compatible with other languages, you just need to acknowledge that exceptions exist and handle them appropriately. The easiest way would be for the tool vendors to implement ModelicaError as an extern "C" function (thus keeping the interface in C) with a single throw statement in a C++ source file. The code calling external functions would need to go through a level of function calls including a try..catch statement, which honestly you should do anyway because what if the external function you're calling is not written in C but in a language that throws an exception? You don't want the Modelica tool to crash altogether, do you? The exception handling, at least on Linux, is the same for all languages, so this approach would do the right thing not just for C++ but even for external functions written in languages such as Ada or Rust (the first language has exceptions, the second uses the exception runtime to implement panics last time I checked).
Motion of order: the discussion about exception handling and ModelicaError is very interesting and important for future development of Modelica, but it is out of scope here, I would warmly invite @fedetftpolimi to make a proposal in a separate ticket and discuss it there 😃
The topic of this ticket is how to solve handle ModelicaError and other ModelicaUtilities.h functions in external objects that are implemented as shared libraries. In fact, I jumped in almost immediately with a related problem, namely handling these functions also in external functions that have no direct access to instantiated external objects. I think it would be valuable if we could find a common solutions to these two problems.
I had some discussion with @henrikt-ma and @t-sommer about the issue that I raised with my MWE. This could be solved by providing a constant external object instance in ModelicaServices, besides the definition of such external object.
within ModelicaServices;
package ModelicaUtilities "Provides external objects to access ModelicaUtilities.h functions"
class Callbacks "External object with pointers to ModelicaUtilities.h functions"
extends ExternalObject;
function constructor
// the implementation is provided by the tool developer, not specified in the MA ModelicaServices
end constructor;
function destructor
// ditto
end destructor;
end Callbacks;
constant Callbacks callbacks = Callbacks() "To be used in state-less functions not tied to external objects";
end ModelicaUtilities;
The pointer to these Callbacks object should have a structure like the one proposed by t-sommer in his first post, or possibly, even better, following the same pattern that is used in FMI for the same purposes, see https://github.com/modelica/fmi-standard/blob/v2.0.x/headers/fmi2FunctionTypes.h#L119.
As suggested by t-sommer at the beginning of this post, the Callbacks external object could be used in other external objects to provide access to ModelicaUtilities.h functions. On the other hand, external functions used outside external objects could get the pointer to ModelicaUtilities.h functions by means of the package constant defined in ModelicaServices, e.g.:
package MWE
partial function baseFunction
input Real x;
output Real y;
end baseFunction;
function function1
extends baseFunction;
algorithm
y := 2*x;
end function1;
function function2
extends baseFunction;
function helper
input Real x;
input ModelicaServices.ModelicaUtilities.Callbacks callbacks;
output Real y;
external "C";
end helper;
y = helper(x, ModelicaServices.ModelicaUtilities.callbacks);
end function2;
function function3 // nice to have, but not strictly necessary
extends baseFunction;
external "C"
y = ext_function2(x, ModelicaServices.ModelicaUtilities.callbacks);
end function2;
model M
replaceable function f = function1 constrainedby baseFunction;
Real y = f(time);
end M;
end MWE;
There would be two ways to do that. One, as in function2(), is via an embedded helper function, the other, as in function3(), by directly passing the package constant to the external C function.
This proposal can be made into a PR to the ModelicaServices library. To complete it, we will also need a minor addition to the ModelicaSpecification Section 12.9.6, to specify the C types of the structure returned by the Callbacks external object. That is actually not a change to the Modelica language, as it has no impact on Modelica compilers, which just need to handle (void *) pointers. It is required to define a tool-independent interface for those callback functions, to be used by whomever writes the C code that must call them. The MLS is the only place where we can define those interfaces, now that the ModelicaUtilities.h file has been removed from the MSL in #3867.
Except:
- Having an object is complicated in terms of future compatibility. There are ways to handle it, but it requires more.
- Using
void*as for external objects is just asking for trouble, seriously stop using it unless needed (and it isn't needed here). - Just having a struct doesn't fully solve the issue as there are more parts to callbacks; and with
void*you are likely to hide that with bad consequences.
But, yes, it will be discussed.
There's still one question I have. Once we have evidenced the need for the constant external object callbacks, why can't we pass this one also to the constructor of other external objects, not just to external functions?
Is there even a point in creating multiple instances of the Callbacks class? Wouldn't all those instance contain the same pointers and be a waste of memory?
I've created an example library to demonstrate the implementation. It already works in Dymola and OpenModelica.
@fedetftpolimi had a quite good idea, i.e. to combine @t-sommer's concept of an external callback object with his idea of importing symbols. This combination has many nice features:
- it is easy to use for external function implementors, which just need to pass a constant external
ModelicaServices.ModelicaUtilities.callbackobject reference to their own external functions to get access to all the ModelicaUtilities functions - it shields the wizardry of symbol importing in the implementation of the constant external object
- it works both at compile time and at run time out of the box
- a reference implementation can be provided in the MSL, though of course tool vendors are free to replace it with their own
- it requires minimal effort for its support by tool vendors, basically just exporting the symbols of the ModelicaUtilities functions both from the compiler and from the runtime, if they use the reference implementation
@fedetftpolimi is implementing a prototype, that could become a reference implementation, and testing it with OpenModelica, see https://github.com/fedetftpolimi/ModelicaUtilities/tree/CallbacksAsConstant. If we see that it works, we can report it here and use it as a basis for a refined proposal. The prototype will work out of the box in any Modelica tool that exports the symbols of ModelicaUtilities function from both the compiler/GUI and the simulation runtime.
I will investigate if it is possible to do this without external object.
Specifically:
- Generic ModelicaUtilities.h for use in projects; that can be used without any additional changes of the library.
- Only add a specific include when using it; no need to construct external objects etc. (That part could be moved to tool-vendors.)
Or simply put: it will just work seamlessly.
If one really wants to be safe there's not even a requirement that the Modelica tool exports the symbols in ModelicaUtilities.h - only that it makes them callable.
We already did that research, and so far the issue of "just including ModelicaUtilities.h" is that it breaks when doing so from within a shared library (DLL) on Windows. I'm not a super-expert in that OS, but so far the only solution to force the Windows runtime linker to resolve symbols to the ModelicaUtilities functions is this piece of code https://github.com/fedetftpolimi/ModelicaUtilities/blob/CallbacksAsConstant/CallbacksImplementation/importer.h that works but is obscure and low-level. Not something you'd want a third party library developer to understand just to be able to call ModelicaError and such from their code.
That's why I jumped in at @t-sommer idea of the external object, as it would hide that low-level code behind a completely hidden tool-provided library that third-party libraries don't have to link against (otherwise we'd be back to making it impossible to make a third-party Modelica library distributed together with a shared library that works with all tools).
We already did that research, and so far the issue of "just including ModelicaUtilities.h" is that it breaks when doing so from within a shared library (DLL) on Windows. I'm not a super-expert in that OS, but so far the only solution to force the Windows runtime linker to resolve symbols to the ModelicaUtilities functions is this piece of code https://github.com/fedetftpolimi/ModelicaUtilities/blob/CallbacksAsConstant/CallbacksImplementation/importer.h that works but is obscure and low-level.
There should be better of doing that.
On Linux when building a shared library, by default by just providing the function declaration you can call a function not defined anywhere. At the linking stage when the shared object is built, it will be listed as an undefined reference without the need to tell where it can be found (that is in which other shared object or executable it is, as this is key to achieve tool vendor independence of third party libraries). When the shared object is loaded, then the runtime linker will try to find it and fail to load it if it can't.
On Mac OS it is possible, but not the default. See this line I added to the ExternalMedia build system for how to do it: https://github.com/modelica-3rdparty/ExternalMedia/blob/master/Projects/CMakeLists.txt#L197
On Windows, both me and @mahge who knows about Windows more than me had to resort to basically doing the symbol lookup in code using the Windows API. I don't exclude there's an easier way, but until someone posts a working proof of concept implementation I'll keep assuming there isn't.
Apparently this proposal of using an external object is not trivial to implement in OpenModelica because it currently lacks support for instantiating external objects at compile time, see https://github.com/OpenModelica/OpenModelica/issues/13044
I underline the issue is in the external object and not in the idea of loading symbols at run-time, as proved by the fact that my MWE https://github.com/user-attachments/files/15516624/mwe.zip uses the same importer code without using an external object and already works with OpenModelica even at compile time.
I'd be curious to see if @HansOlsson can provide a simpler MWE to resolve ModelicaUtilities functions's symbols in Windows from a DLL, as that could open the possibility to 'just #include "ModelicaUtilties.h"' without the need for an external object.
Apparently this proposal of using an external object is not trivial to implement in OpenModelica because it currently lacks support for instantiating external objects at compile time, see OpenModelica/OpenModelica#13044
We are working on it 😃
At the moment, the best solution I see is to have a constant external object defined in ModelicaServices, which all external functions and object can use by simply referring to it, ModelicaServices.ModelicaUtilities.callbacks. At the end of the day, this ModelicaServices.ModelicaUtilities.callbacks constant is nothing but a pointer to a struct that contains the pointers to all the ModelicaUtilities functions.
Now, the whole point (and crucial issue) of this proposal is that it should work seamlessly both during compile time and simulation time. Implementing external object handling at compile time in general is a non-trivial task, but just implementing handling of constant external objects, which are nothing but a pointer to something that is obtained by calling the constructor, seems much easier to me.
Please give us some time to figure this out and test it properly.