Add dynamic validation of FMUs

[t-sommer]

• load all API functions
• get directional derivatives
• handling of illegal VRs, min / max values
• get / set state

[masoud-najafi]

A minimal test on fmi2SetDebugLogging

[nickbattle]

Many thanks for the FMPy walkthrough today.

I was particularly interested to see how you did the XML validation (and perhaps how you could do dynamic API validation), comparing this to our own formal VDM models and the VDMCheck tool.

But it sounds like we are trying to do different things. FMPy is taking a pragmatic view and validating aspects of the XML that are known to cause people problems - I think you questioned the value of checking every little thing that could be wrong with the XML, if these things do not cause problems. The VDM model on the other hand is attempting to formally define the static (XML) and dynamic (API) behaviour that is given informally in the FMI Standard. So every little thing matters!

It was good (in a sense) to hear that you know that >50% of the FMUs in the cross-check repository fail static validation. We agree, and the problems we find are in the same areas you mentioned (model structures, missing units and illegal structured names).

We also share your frustration of not being able to easily refer a validation rule back to the FMI Standard. Currently, VDMCheck just gives a section number and a one-line explanation. But it is not always obvious what the rules are or where they are defined.

So how can the formal model help FMPy ...?

If you think that our list of static validation checks would be worth reviewing to see whether any of them are useful to add to FMPy, I'd be happy to explain the formal model.

You may also be interested to review the dynamic validation checks that we perform for each API, as these might also be worth adding to FMPy. For example, fmi2SetReal checks that:

• the vrs, nvr and values arrays are consistent sizes
• that the general rules for setting the variables are met (ie. the SEA, INI, INC etc rules about which variables can be set)
• the vrs must all exist and refer to Real variables
• the values being set must be within the min/max for the variables (possibly after inheritance from their defined type)
• you cannot set the independent variable.

If all that succeeds, then you can set the values - this is also defined formally in terms of an abstract state, but that is probably less useful for FMPy.

Lastly a caveat: although we're trying to produce a formal model that is correct, it is not definitive, so you may find that we have interpreted the FMI Standard differently to you. If you find such differences, they are important to resolve, so please let us know!

[t-sommer]

I think you questioned the value of checking every little thing that could be wrong with the XML, if these things do not cause problems.

Not at all. It just defines the order in which I implement the checks. I see VDMCheck (and other tools) as complementary. The more tools we have the better. I'll be happy to get back to you ;-)

[t-sommer]

You may also be interested to review the dynamic validation checks that we perform for each API, as these might also be worth adding to FMPy. For example, fmi2SetReal checks that:

• the vrs, nvr and values arrays are consistent sizes
• that the general rules for setting the variables are met (ie. the SEA, INI, INC etc rules about which variables can be set)
• the vrs must all exist and refer to Real variables
• the values being set must be within the min/max for the variables (possibly after inheritance from their defined type)
• you cannot set the independent variable.

These check are quite expensive if you perform them every time. I think the calling sequence of an importer (like FMPy itself) is better checked with the Reference FMUs.

[nickbattle]

These check are quite expensive if you perform them every time.

Yes, of course. But the formal model is trying to define what the rules are, rather than producing something that is efficient.

[clagms]

This is a very good initiative @t-sommer !

These check are quite expensive if you perform them every time. I think the calling sequence of an importer (like FMPy itself) is better checked with the Reference FMUs.

Maybe I misunderstood the goal. Isn't the goal to define dynamic check that can be turned on optionally? Or some kind of testing framework that loads an FMU and runs all the checks? If that's the case, then performance is not such a big deal (unless certain really long co-simulations crash and you need to rerun them with all checks enabled to see if it was a conformance issue).

We could probably pass a fine comb through the document and elicit all the dynamic rules we could check. One that immediately comes to mind is whether the FMU uses feedthrough or now (at least a warning can be given if using, just to inform the co-simulation user that the numerical results may be off because of that).

Maybe since this is some kind of ongoing development, it would be nice to have a library on top of FMPy that accepts some "higher level dynamic rules" so that we can just add rules over time for checks that make sense...

[trulede]

I use FMPy for running FMUs. Its great. Already when you include the dependencies (e.g. numpy) FMPy is becoming a fairly large package. Having a validator is nice, however having it as a separate Package or SubPackage, so I don't need to install it ... would be even nicer.

It's only necessary to validate an FMU once, and that can be done in a separate workflow to actually running FMUs with FMPy.

[t-sommer]

We're currently testing an online version of the FMU Check on https://fmi-standard.org/fmu-check/ that's powered by FMPy. Maybe this could also be an option if you don't want to install FMPy locally.

[trulede]

Thanks. I install FMPy into a Docker image and actually only use only the FMI classes. I've been looking though the code and think it could be possible to define some kind of minimal or core sub-package with only the FMI related code. It's a bit tricky with some code in the init.py, but if there is a way to do it and not affect the rest of the code base ... could be nice.

[t-sommer]

@trulede, you might want to have a look at the discussion in https://github.com/conda-forge/fmpy-feedstock/issues/11

[trulede]

Thanks @t-sommer I will try a few ideas out and get back to you (perhaps with a PR).