Clarify/Allow ExternalObject (member) functions to be used for (flexible) array parameters/dimensions

[beutlich]

Based on the technical blog post from @Claytex, it is currently either tool-specific or unspecified, if ExternalObject (member) functions can be used for flexible array sizes (as described in section 12.4.5 of the Modelica Language Specification v3.4).

The below example demonstrates what @GarronFish's intention is, i.e., to read a vector of unknown dimension from an XML file and use it as parameter.

// Fails in Dymola 2020/2020x/2021/2021x
// Works in SimulationX 4.x
model SizingExampleXML2
  constant String hString = "multiTank.tankHeights" "XML element name";
  constant String fName = "C:\\temp\\multiTank1.xml" "XML file name";
  parameter ExternData.XMLFile extObj(fileName=fName) "Implemented as external object";
  parameter Real heightVector[:] = extObj.getRealArray1D(hString, extObj.getArraySize1D(hString)) "Array parameter";
  annotation(uses(ExternData(version="2.5.0")));
end SizingExampleXML2;

with

<?xml version="1.0"?> <!-- multiTank1.xml -->
<ExternData>
  <multiTank>
     <tankHeights>{30,20,10,0}</tankHeights>
     <initialTankLevels>{10,0,0,0}</initialTankLevels>
  </multiTank>
</ExternData>

This example fails in Dymola 2020 with

Translation of SizingExampleXML2:

Failed to expand the variable heightVector
and its definition equation:
extObj.getRealArray1D_Unique'"extObj"'(
hString, 
extObj.getArraySize1D_Unique'"extObj"'(
hString, 
extObj.xml), 
extObj.xml)

Errors detected in variable declarations.

Translation aborted.

ERRORS have been issued.

but runs in SimulationX 4.0.

@GarronFish's workaround was to wrap the C code of the external object in another C functions to be used as Modelica functions for flexible array sizes. (In his case, problems with thread-safety were introduced when using global variables for one-time construction and destruction of the external object. This of course could be improved, but I'd claim, that it would be even better, if no such workaround would be necessary at all.)

Please clarify, if ExternalObject (member) functions can be used for parameter expressions of flexible array size.

[beutlich]

@HansOlsson Any feedback please? Thanks.

[HansOlsson]

@HansOlsson Any feedback please? Thanks.

Just some quick feedback (will try to more completely review Modelica issues in a bit).

To me it seems odd that an ExternalObject object in the model is used to evaluate a structural parameter, as that implies that the ExternalObject functions (that may have side-effects and/or modifying the ExternalObject) are called outside of the model evaluation.

[beutlich]

In the presented case the external object only is used as cache for the read/parsed file. The workaround would be to use external functions only and hide the caching in global variables together with thread-safe access. To me this looks similiarly as the old Dymola tables implementation (of MSL <= 3.2.1) compared to the existing implementation based on external objects with dedicated construction and destruction. Additionally in MSL, we cache the opened files for reading in ModelicaInternal implementation.

[HansOlsson]

In the presented case the external object only is used as cache for the read/parsed file. The workaround would be to use external functions only and hide the caching in global variables together with thread-safe access.

The problem is that if you see:

   MyExternalObject m=MyExternalObject(...);
   parameter Real arr[2,2]=SomeFun(m, ...);

you don't know whether calling SomeFun modifies the external object or if it depends on something truly external. But I guess you could include it and an assertion for checking if it had been modified.

If that is hidden in a function you can be sure that any modification of the external object is intended to be lost.

   function foo
      ...
      output Real arr[2,2];
   protected
     MyExternalObject m=MyExternalObject(...);
    algorithm
       arr=SomeFun(m, ...);
    end foo;
    parameter Real arr[2,2]=foo(...);

[beutlich]

you don't know whether calling SomeFun modifies the external object or if it depends on something truly external

Can't we use pure/impure to indicate these cases?

[HansOlsson]

you don't know whether calling SomeFun modifies the external object or if it depends on something truly external

Can't we use pure/impure to indicate these cases?

Possibly. We might need to ensure that impure is sufficiently clear for functions taking external objects.

[beutlich]

If that is hidden in a function you can be sure that any modification of the external object is intended to be lost.

Does Dymola call the destructor directly after calling foo?

[HansOlsson]

If that is hidden in a function you can be sure that any modification of the external object is intended to be lost.

Does Dymola call the destructor directly after calling foo?

At the end of foo as I recall.

[HansOlsson]

As far I can see it is already stated that tools don't have to translate these. https://specification.modelica.org/master/introduction1.html#scope-of-the-specification

In order to allow this structural analysis, a tool may reject simulating a model if parameters cannot be evaluated during translation – due to calls of external functions or initial equations/initial algorithms for fixed=false parameters. Accepting such models is a quality of implementation issue.

[beutlich]

Let's assume the external object construction of MyExternalObject takes a long time and you need to call the function foo many times with (different) inputs. How to optimize the simulation performance? I guess you can cache the constructed object, but this leaves the question when to invalidate the cache? I can only think of cache invalidation during the module destruction (assuming shared object like an FMU). Is this really the way we want to progagate to library developers? To me this sounds like a workship solution only.

[HansOlsson]

Let's assume the external object construction of MyExternalObject takes a long time and you need to call the function foo many times with (different) inputs.

To clarify:

My conclusion is that the current specification states neither of the variants need to be translated, as both involve external functions.

So it is an enhancement; not something unclear.

[beutlich]

@HansOlsson I just noticed that the milestone was moved. Back to my original issue and my repeated tries to make it somehow working in Dymola. You mentioned example foo returning an array with fixed dimensions 2x2.

   function foo
      ...
      output Real arr[2,2];
   protected
     MyExternalObject m=MyExternalObject(...);
    algorithm
       arr=SomeFun(m, ...);
    end foo;
    parameter Real arr[2,2]=foo(...);

But what the customer needs are dynamic array sizes read from file and used as parameters. However, if these dimensions are inputs to the function, I still do not get it running in Dymola 2021x.

// Fails in Dymola 2021x
model SizingExampleXML3
  constant String hString = "multiTank.tankHeights" "XML element name";
  constant String fName = "C:\\temp\\multiTank1.xml" "XML file name";
  final parameter Integer n = readArraySize1D(hString, fName) "Array size";
  parameter Real heightVector[n] = readRealArray1D(hString, fName, n) "Array parameter";
  function readArraySize1D "Read array size"
    input String hString "XML element name";
    input String fName "XML file name";
    output Integer n "Array size";
  protected
    ExternData.Types.ExternXMLFile extObj = ExternData.Types.ExternXMLFile(fName, verboseRead=true) "External XML file object";
  algorithm
    n := ExternData.Functions.XML.getArraySize1D(hString, extObj);
  end readArraySize1D;
  function readRealArray1D "Read array"
    input String hString "XML element name";
    input String fName "XML file name";
    input Integer n "Array size";
    output Real arr[n] "Array";
  protected
    ExternData.Types.ExternXMLFile extObj = ExternData.Types.ExternXMLFile(fName, verboseRead=true) "External XML file object";
  algorithm
    arr := ExternData.Functions.XML.getRealArray1D(hString, n, extObj);
  end readRealArray1D;
  annotation(uses(ExternData(version="2.6.1")));
end SizingExampleXML3;

Edit: See also #1811 and https://github.com/modelica/ModelicaStandardLibrary/issues/1856#issuecomment-272677293.

2nd Edit: Applying annotation(__Dymola_translate=true); to function readArraySize1D resolves the issue.

[beutlich]

Citing myself from ^^

But what the customer needs are dynamic array sizes read from file and used as parameters.

It seems that Dymola 2025x users will benefit from a new tool-specific annotation __Dymola_UnknownArray according to this recent Linkedin post by @DagBruck. That is one good step ahead towards some better and hopefully standarized way in supporting flexible array parameters. Thanks.