Breaking ModelicaServices cyclic dependency on Modelica

[sjoelund]

Currently, ModelicaServices is backwards compatible with older versions of itself and has noneFromVersion conversion annotations. However, it has a uses-annotation on Modelica which in turn has a uses-annotation on ModelicaServices. This means that it is not possible to have for example Modelica 3.2.2 loaded with ModelicaServices 3.2.3 or 4.0.0. There are some interfaces in MSL that are used as well as some icons, but I believe these would work just as well if they were put into ModelicaServices and if the uses-annotation was removed, it would instantly be possible to load a more recent ModelicaServices if needed. The disadvantage would be that some form of sanity check from MSL's point of view would be lost and it would need to rely on ModelicaServices version/conversion annotations being correct.

This might for example make it possible for a Modelica tool vendor to ship only 1 version of ModelicaServices that is compatible with all previous MSL versions using ModelicaServices. It would also make package management easier.

I should note that I have not tested such changes, but would like a discussion to see if this would be something we are interested in changing (and would perform tests on any pull request I create for this).

[HansOlsson]

In parts that seems like a good idea, and tool vendors may already be able to that in the versions they provide.

The question is more whether we should do that in the "standard" version, or just have it as a suggestion.

Note that tool implementations normally also add implementations, and in some cases those may depend on other classes in MSL that may or may not be changed between versions; even something as innocent as Position was moved between 3.2.3 and 4.0.0, a tool may circumvent that by "reimplementing" it in ModelicaServices (a Real with unit="m").

Thus it might be simpler to just add that a tool may do it, possibly with the comment above. (I can't remember if we already tried that in Dymola, and whether there were any issues or not.)

[sjoelund]

From my point of view it would be nicer to have it in the default version. I suppose that's because we script most of the changes to services. It's just a small patch + some commands:

https://github.com/OpenModelica/OpenModelica-ModelicaStandardLibrary/blob/OM/maint/4.0.x/ModelicaServices.patch

setClassComment(ModelicaServices, "ModelicaServices (OpenModelica implementation) - Models and functions used in the Modelica Standard Library requiring a tool specific implementation");
setComponentModifierValue(ModelicaServices, target, $Code(="OpenModelica"));
setComponentModifierValue(ModelicaServices.Machine, Integer_inf, $Code(=OpenModelica.Internal.Architecture.integerMax()));
addClassAnnotation(ModelicaServices.ExternalReferences.loadResource, annotate=__OpenModelica_EarlyInline(true));

One could also consider having some PartialModelicaServices package if that part seems too big, and let it contain the things that vendors are not supposed to change. Because, yes... It would need to have its own types for those common visualizers. That way you could also ensure that the interface is adhered to.

[sjoelund]

Thus it might be simpler to just add that a tool may do it, possibly with the comment above. (I can't remember if we already tried that in Dymola, and whether there were any issues or not.)

I think that you then might end up with models from that tool having annotations like uses(Modelica(version="3.2.3"), ModelicaServices(version="4.0.0")), which other tools would not be able to handle. We have seen models like that for various reasons.

[hubertus65]

I'd prefer not to have cyclic dependencies, but I have no strong opinion on how to solve them. Your suggestion looks pragmatic to me.

[henrikt-ma]

Maybe this is a good time to ask ourselves whether it really is a good idea that ModelicaServices is tied to the MSL rather than the MLS? If we could reorganize things so that ModelicaServices was part of the language specification, there wouldn't be any question about uses annotations (it would be sufficient for the MSL to express which language version it requires, and the reorganization should have removed all current dependencies of ModelicaServices on the MSL).

I don't know enough about Animation to be able to tell whether such a reorganization would be possible, but maybe @otronarp has thought about this in the System Modeler context?

[sjoelund]

The problem with tying it to MLS would be the ability to update the functionality when necessary. It's probably not a huge problem given that since 2017 (MLS 3.4), the only changes were to add System.exit and the Vector visualizer to the package.

But I don't know what would be gained by having it in the specification. Some parts have more to do with interfacing with Modelica tools than anything interesting in the specification.

The URI resolving and perhaps the machine constants would fit as built-in operators in the language. And the System.exit, visualizers and probably the choices of solvers belong more to the tools (and solver choices perhaps to specific versions of a tool).

[maltelenz]

Some parts have more to do with interfacing with Modelica tools than anything interesting in the specification.

Isn't what Modelica tools do with Modelica libraries exactly what the specification is about?

In principle I agree with Henrik that not having a ModelicaServices and replacing it with features in the MLS is a good goal. This way any new features go through the proper language design steps, and they will also be made available to other libraries that may want to use them.

[sjoelund]

I believe the problem was that standardization took too long and the language already mostly supported what was needed, so ModelicaServices was created. The specification even mentions ModelicaServices, for example:

A predefined type ModelicaServices.Types.SolverMethod defines the methods supported by the respective tool by using the choices annotation.

Anyway, I have tested that the following works . It's still wrong as it uses Modelica from within ModelicaServices, but it shows that both MSL 3/4 actually work with the same code (note that it uses OpenModelica-specific changes so all examples work properly).

package ModelicaServices
  constant String target= "OpenModelica";

  package Animation
    model Shape
      extends Modelica.Utilities.Internal.PartialModelicaServices.Animation.PartialShape;
    end Shape;

    model Surface
      extends Modelica.Utilities.Internal.PartialModelicaServices.Animation.PartialSurface;
    end Surface;

    model Vector
      extends Modelica.Utilities.Internal.PartialModelicaServices.Animation.PartialVector;
    end Vector;
  end Animation;

  package ExternalReferences
    function loadResource
      extends Modelica.Utilities.Internal.PartialModelicaServices.ExternalReferences.PartialLoadResource;
      algorithm
      fileReference:=OpenModelica.Scripting.uriToFilename(uri);
      annotation (__OpenModelica_EarlyInline = true);
    end loadResource;
  end ExternalReferences;

  package Machine
    final constant Real eps=1e-15;
    final constant Real small=1e-60;
    final constant Real inf=1e60;
    final constant Integer Integer_inf=OpenModelica.Internal.Architecture.integerMax();
  end Machine;

  package System "System dependent functions"
    impure function exit
      extends Modelica.Utilities.Internal.PartialModelicaServices.System.exitBase;
      external "C" exit(status) annotation(Include="#include <stdlib.h>", Library="ModelicaExternalC");
    end exit;
  end System;

  package Types
    type SolverMethod = String
      "String defining the integration method to solve differential equations in a clocked discretized continuous-time partition"
      annotation (choices(
        choice="External" "Solver specified externally",
        choice="ExplicitEuler" "Explicit Euler method (order 1)",
        choice="ExplicitMidPoint2" "Explicit mid point rule (order 2)",
        choice="ExplicitRungeKutta4" "Explicit Runge-Kutta method (order 4)",
        choice="ImplicitEuler" "Implicit Euler method (order 1)",
        choice="ImplicitTrapezoid" "Implicit trapezoid rule (order 2)"));
  end Types;

  annotation (
    version="4.0.0",
    conversion(
      noneFromVersion="1.0",
      noneFromVersion="1.1",
      noneFromVersion="1.2",
      noneFromVersion="3.2.1",
      noneFromVersion="3.2.2",
      noneFromVersion="3.2.3"));
end ModelicaServices;

[sjoelund]

So I created a draft PR #3912. I thought it was nicest to create a separate package ModelicaAnimationInterface (or whatever we want to call it), with the animation parts in it. Where to actually put icons or if we need them in ModelicaServices / ModelicaAnimationInterface is another question. As is what parts of MSL to remove or extend from the other packages. I did move some icons to ModelicaAnimationInterface so it is stand-alone, but it looks awkward to have MSL extend from icons in it. So maybe some icons should be duplicated.

@mtiller has had some ideas for how to better structure packages in the past that would have been handy now. A ModelicaIcons package could be useful to have, but then you might get more and more packages visible in the tools...

[HansOlsson]

Hmm... To me this is starting to get too complicated.

I have no problem with a tool vendor modifying ModelicaServices in this way, but using it in MSL does not seem good.

In particular I see problem with the duplicated types for Position etc (and we clearly don't want to move them from MSL); as we have a number of GUI-operations that rely on propagating variables and this might cause these new internal types to spread.

[sjoelund]

Duplicating Position is probably not a problem given that it's such a simple type and has no annotations on it. So it wouldn't need to be an alias. Most of these changes would be to the MSL so that tool vendors would not need to make such extensive changes. In particular, if you see the PR, these types were moved to ModelicaAnimationInterface, which is not intended to be changed by vendors. ModelicaServices would actually be very similar to before (I mainly changed exit/URI to not use a partial version in MSL https://github.com/modelica/ModelicaStandardLibrary/blob/4fe31b146867f93efc620d832d40aedd1702bed6/ModelicaServices/package.mo, and moved a lot of MSL code to ModelicaAnimationInterface).

The nullRotation move could also be avoided by just changing the modifiers in that class.

A nice solution would be to have:

• Modelica, which uses ModelicaServices, which uses ModelicaAnimationInterface, which uses ModelicaIcons

It would be maintainable, but has the disadvantage that we have more top-level packages. Only ModelicaServices would be modified by vendors.

[maltelenz]

I'm strongly against cluttering the top level class list with more things most users will not be interested in ever looking at or using directly. It's bad enough that we already have ModelicaServices and Complex, both of which would better live in the Modelica language (or I guess inside Modelica. for Complex if we don't want complex numbers in the language).

[sjoelund]

I'm strongly against cluttering the top level class list with more things most users will not be interested in ever looking at or using directly. It's bad enough that we already have ModelicaServices and Complex, both of which would better live in the Modelica language (or I guess inside Modelica. for Complex if we don't want complex numbers in the language).

The alternative would be to move things that should not be modified by the tool vendor into ModelicaServices. Or duplicating a lot of code. Or not using for example Icons in ModelicaServices. Removing the use of some of the visualizers would not be possible though. If we want to be able to use MSL 3.2.3 or older together with a newer ModelicaServices, it needs to have everything available without referring to MSL itself.

Additional top-level packages could be hidden if necessary, and still visible in the GUI for example if you have:

within Modelica;
package Icons = ModelicaIcons;

Myself, I do like Complex.mo. It's a small package and easy to use without MSL.

As for adding more things to the language itself, this could be a separate issue to simplify ModelicaServices in the future.

[maltelenz]

Shipping one ModelicaServices per version is a negligible effort for us, while adding more top level packages adds unnecessary clutter and complexity for every single end user of our tool. Therefore I'm strongly against the proposed change.

[sjoelund]

It's not just about shipping one ModelicaServices. It's about being able to use a different ModelicaServices than the MSL version. Right now they both have uses-annotations that forces both of them to be the same. This is quite annoying since you need to unload both MSL and ModelicaServices if you want to change libraries even though ModelicaServices itself is actually backwards-compatible with older versions (even ModelicaServices 4 to 3). And if you load Modelica 3.2.3 it has uses on ModelicaServices 3.2.3. And ModelicaServices 4.0.0 says it doesn't require a conversion script to be used, so a tool might think that loading it is a good idea. But then it has a uses-annotation that is not compatible.

Whenever you have cycles in the dependencies, you get these issues. The 2 packages are so tied into each other that they really should be 1 single library. Or be properly split up.

adding more top level packages adds unnecessary clutter and complexity for every single end user of our tool.

Dymola does not show Complex or ModelicaServices (adding Protection(access=Access.hide) to libraries not intended for users to show). We could hide these for the user.

[maltelenz]

It's not just about shipping one ModelicaServices. It's about being able to use a different ModelicaServices than the MSL version. Right now they both have uses-annotations that forces both of them to be the same. This is quite annoying since you need to unload both MSL and ModelicaServices if you want to change libraries even though ModelicaServices itself is actually backwards-compatible with older versions (even ModelicaServices 4 to 3). And if you load Modelica 3.2.3 it has uses on ModelicaServices 3.2.3. And ModelicaServices 4.0.0 says it doesn't require a conversion script to be used, so a tool might think that loading it is a good idea. But then it has a uses-annotation that is not compatible.

We solve this issue for our users by providing a "Change MSL version" when right-clicking on the loaded one. It takes care of all the gritty details.

Whenever you have cycles in the dependencies, you get these issues. The 2 packages are so tied into each other that they really should be 1 single library. Or be properly split up.

Right, they should be 1 single library, Modelica. If it needs anything implemented by tools, it should be specified in the language.

I haven't looked in detail at what the interdependencies actually are, but I'm more in favor of some duplication of code than I am more top level classes, if that is what is needed to break the circle.

Dymola does not show Complex or ModelicaServices (adding Protection(access=Access.hide) to libraries not intended for users to show). We could hide these for the user.

We used to do this as well, but it is just a hack with its own problems. It makes the Modelica library have "magic" loose ends that a user cannot understand or look at, but "somehow" work. Not something to aspire to for a standard library, which should set an example in how to create good libraries.

[sjoelund]

I haven't looked in detail at what the interdependencies actually are, but I'm more in favor of some duplication of code than I am more top level classes, if that is what is needed to break the circle.

Even if it means adding things to ModelicaServices that tool vendors are not supposed to modify?

[maltelenz]

Even if it means adding things to ModelicaServices that tool vendors are not supposed to modify?

Yes. I don't see this as a big issue, if each piece (class?) is clearly marked if intended to be modified by tool vendors or not.

[HansOlsson]

Duplicating Position is probably not a problem given that it's such a simple type and has no annotations on it. So it wouldn't need to be an alias.

My concern wasn't about maintaining that, but that if you used the animation-parts in models and propagated parameters the user model would contain ModelicaAnimationServices.Types.Position instead of Modelica.Units.SI.Position which would be non-ideal for users.

[sjoelund]

My concern wasn't about maintaining that, but that if you used the animation-parts in models and propagated parameters the user model would contain ModelicaAnimationServices.Types.Position instead of Modelica.Units.SI.Position which would be non-ideal for users.

How would you prefer it, then? Using Real with modifiers directly inside ModelicaServices?

[casella]

My comments on this issue, trying to reach a possible consensus?

1. We are rolling out a package management system in OpenModelica, but the current cyclic dependency between Modelica and ModelicaServices is causing us a lot of trouble which is totally unnecessary; @hubertus65 also agrees that removing it would be nice. In fact, not having cyclic dependencies among packages makes implementing package management systems much easier, which I believe is a plus for all tool vendors. I'm not aware of any other libraries that have them.
2. There is actually no need to have multiple versions of ModelicaServices, now or for the foreseeable future - a properly designed one which is self-contained and has no dependencies would do. Types for position parameters in animations can use Real(unit = "m") without creating cyclic dependencies on the MSL (and different versions thereof) or duplicate types.
3. I see the point of making ModelicaServices actually part of the MLS, but at 338 pages that document is probably already long enough. In fact, there were talks in the past to develop Modelica 4 in a way that moved parts (most?) of the specification into libraries, which seemed to me a very good design principle. We never managed to do that, but at least I would avoid going in the opposite direction. ModelicaServices seems to me a good pragmatic compromise.
4. Even though I see the point of splitting ModelicaServices in multiple libraries, I agree with @maltelenz that we should not further clutter the top-level namespace with ModelicaAnimationInterface or other further packages, so I would keep one ModelicaServices package, just removing the cyclic dependencies, even though this may be slightly sub-optimal in terms of maintenance.

I am personally in favour of the simplest possible solution that requires the minimum amount of changes to the status quo, while resolving the cyclic dependency issue, which is what really hurts, and is the reason why this ticket was opened. Other further improvements may be discussed for the long term, but please let's do that in a separate ticket. I would try to solve this specific issue with a solution that fits everybody and makes a (small) step forward.

[maltelenz]

1. In fact, not having cyclic dependencies among packages makes implementing package management systems much easier, which I believe is a plus for all tool vendors. I'm not aware of any other libraries that have them.

With the risk of derailing this issue completely. If the specification allows cyclic dependencies (which it clearly seems to do), wouldn't any good package management system need to handle them anyway? If we don't want to forbid cyclic dependencies in the language, isn't this just removing a symptom of something that needs to be implemented correctly (detecting cycles)?

[sjoelund]

With the risk of derailing this issue completely. If the specification allows cyclic dependencies (which it clearly seems to do), wouldn't any good package management system need to handle them anyway? If we don't want to forbid cyclic dependencies in the language, isn't this just removing a symptom of something that needs to be implemented correctly (detecting cycles)?

OpenModelica does handle cycles. But cycles have another problem. If we require ModelicaServices to use a particular version of Modelica and Modelica a particular version of ModelicaServices, neither of them is actually backwards-compatible. If any library out there has a uses(Modelica(version="3.2.2"), ModelicaServices(version="3.2.3")), it cannot really be loaded reliably. If you try to load Modelica 3.2.2 first, you will then also load ModelicaServices 3.2.2 and the second uses won't be satisfied. And you cannot unload ModelicaServices 3.2.2 to load 3.2.3, because that would require unloading Modelica as well.

So it only really works if nobody has uses-annotations on ModelicaServices (which is mostly true, but not universally true).

[HansOlsson]

How would you prefer it, then? Using Real with modifiers directly inside ModelicaServices?

I prefer using the types in MSL; as that it was users would normally use.

As stated before I don't see a compelling reason that Modelica Association should modify ModelicaServices in this way. If OpenModelica want to make their ModelicaServices less version dependent they can do that.

[maltelenz]

If any library out there has a uses(Modelica(version="3.2.2"), ModelicaServices(version="3.2.3")), it cannot really be loaded reliably. If you try to load Modelica 3.2.2 first, you will then also load ModelicaServices 3.2.2 and the second uses won't be satisfied. And you cannot unload ModelicaServices 3.2.2 to load 3.2.3, because that would require unloading Modelica as well.

I see, thank you for the clarification. You still need to handle this situation though (for other libraries, since the language does not forbid it).

I agree with @HansOlsson, the benefits seem marginal. We should focus our efforts on getting rid of ModelicaServices completely instead.

[henrikt-ma]

I agree with @HansOlsson, the benefits seem marginal. We should focus our efforts on getting rid of ModelicaServices completely instead.

But wouldn't removing cyclic dependencies and making the same ModelicaServices usable with multiple MLS versions actually be a very good first step in this direction?

[sjoelund]

How would you prefer it, then? Using Real with modifiers directly inside ModelicaServices?

I prefer using the types in MSL; as that it was users would normally use.

As stated before I don't see a compelling reason that Modelica Association should modify ModelicaServices in this way. If OpenModelica want to make their ModelicaServices less version dependent they can do that.

I do believe if you keep the PartialModelicaServices inside of MSL also, and extend from this before ModelicaServices... If you use the visualizers/etc from Modelica it will probably propagate the Modelica.xxx paths. It's a bit of duplicated code, but might work. I don't use the GUI though, so I'm not sure what things that propagate that you want to keep.

[sjoelund]

But wouldn't removing cyclic dependencies and making the same ModelicaServices usable with multiple MLS versions actually be a very good first step in this direction?

This is my thoughts as well. It would be nicely backwards-compatible but you could remove some references to it from MSL as the language includes these things. And once no more references to ModelicaServices exists, the uses-annotation could be removed.

I think these are quite easy to standardize in some way:

Modelica/Clocked/Types/SolverMethod.mo:type SolverMethod = ModelicaServices.Types.SolverMethod
Modelica/Constants.mo:  final constant Real eps=ModelicaServices.Machine.eps
Modelica/Constants.mo:  final constant Real small=ModelicaServices.Machine.small
Modelica/Constants.mo:  final constant Real inf=ModelicaServices.Machine.inf
Modelica/Constants.mo:  final constant Integer Integer_inf=ModelicaServices.Machine.Integer_inf
Modelica/Utilities/System.mo:  extends ModelicaServices.System.exit;
Modelica/Utilities/Files.mo:     extends ModelicaServices.ExternalReferences.loadResource;

The visualizers are a bit harder to move into the standard:

Modelica/Mechanics/MultiBody/Visualizers/Advanced/Surface.mo:  extends ModelicaServices.AnimationInterface.PartialSurface;
Modelica/Mechanics/MultiBody/Visualizers/Advanced/Surface.mo:  extends ModelicaServices.Animation.Surface;
Modelica/Mechanics/MultiBody/Visualizers/Advanced/Shape.mo:  extends ModelicaServices.Animation.Shape;
Modelica/Mechanics/MultiBody/Visualizers/Advanced/Shape.mo:  extends ModelicaServices.AnimationInterface.PartialShape;
Modelica/Mechanics/MultiBody/Visualizers/Advanced/Vector.mo:  extends ModelicaServices.Animation.Vector;
Modelica/Mechanics/MultiBody/Visualizers/Advanced/Vector.mo:  extends ModelicaServices.AnimationInterface.PartialVector;

One thing I could think of for the visualizers would be some way to have extends, but like a macro version (so the extended elements can access things from the base class). Because part of the problem is we want to have the definitions in Modelica and extend the implementation in ModelicaServices:

model Shape
  extends PartialShape;
  macro extends ModelicaServicesShape; // And have some built-in names in the language for what to extend
end Shape;

model ModelicaServicesShape
  // This kind of structure is not possible and the reason why ModelicaServices depends on Modelica
  // The PR makes this also work if we include exactly the same types/modifiers in here (duplicating all code)... but it's a bit error-prone. The types are the same lexically but might actually point to different things. Multiple inheritance is a little odd in Modelica...
  ExternalObject myToolInterface = /* ... uses things from the PartialShape */;
end ModelicaServicesShape;

But I also think that is orthogonal work and would like to have a nicer ModelicaServices until everything is in place.

[eshmoylova]

In response to @casella's comment and also @hubertus65:

1. We are rolling out a package management system in OpenModelica, but the current cyclic dependency between Modelica and ModelicaServices is causing us a lot of trouble which is totally unnecessary; @hubertus65 also agrees that removing it would be nice. In fact, not having cyclic dependencies among packages makes implementing package management systems much easier, which I believe is a plus for all tool vendors. I'm not aware of any other libraries that have them.

I have seen cyclic dependencies in some vendor libraries in the past. I can provide details here or in a private conversation, if desired. So you may not be able to avoid such things by just fixing ModelicaServices.

[sjoelund]

I have seen cyclic dependencies in some vendor libraries in the past. I can provide details here or in a private conversation, if desired. So you may not be able to avoid such things by just fixing ModelicaServices.

The only other library I have seen with this is ClaRa-ClaRa_Obsolete. But ClaRa actually doesn't even use ClaRa_Obsolete... (It also has uses on ObsoleteModelica4, which it doesn't use either)