Unsuppored nested namelists

[kripnerl]

I am not sure whether it is part of the namelist specification or not, but I met nml file with nested namelists like this:

f90nml.reads("""
&key
 &inner_key_1
  a = 1
  b = 2
 /
 &inner_key_2
  a = 10
  b = 20
 /
/
""")

Namelist([('key', Namelist()),
          ('inner_key_2', Namelist([('a', 10), ('b', 20)]))])

Expected output would be:

Namelist([('key', Namelist([
                               ('inner_key_1', Namelist([('a', 1), ('b', 2)]))
                               ('inner_key_2', Namelist([('a', 10), ('b', 20)]))]))])

or something like this.

I am not sure if this is an issue; however, I find it worthy of mentioning.

[marshallward]

I'm not sure I've ever seen that syntax before, with ~namelists~ namelist group embedded in ~namelists~ a namelist group. Is there a compiler which understands them?

[marshallward]

Reading the standard is always more subtle than it seems, but 13.11.3.1 does say that & (+ the group name) is only to be followed by "a sequence of zero or more name-value subsequences separated by value separators" (i.e. name=value), and terminated by /.

Having said that, standards and reality do not necessarily need to agree, so if there's a compiler that supports this sort of thing I would be interested to know.

[marshallward]

To my surprise, this code appears to at least be doing something one might expect:

integer :: a, b
namelist /inner_key_1/ a, b
namelist /inner_key_2/ a, b
open(10, file="test.nml")
read(10, nml=inner_key_1)
print*, a, b
read(10, nml=inner_key_2)
print*, a, b
end

where test.nml contains the namelist above.

I can't for the life of me figure out how to reference the key group though.

Edit: I used gfortran 10.1 to test this. Intel Fortran 16.0.3 gives similar results.

[marshallward]

I am probing this some more, and it seems that any attempt to read a namelist group named key fails since the parser encounters a & (from &inner_key_1) before encountering the / terminator of &key.

My feeling here is that the parser that is reading test.nml is ignoring all content prior to its ~namelist~ group. For example, when this read() is executed:

read(10, nml=inner_key_1)

it seems to be ignoring all content prior to &inner_key_1, and only starts reading once it gets to this point, and then finally stops once it reaches a /.

That is, the content defined outside of the namelist is ignored for all practical purposes. So it is my guess that &key ... / has no syntactic meaning here.

This starts to touch on certain issues, like whether we should also be applying serialized queries of the file, rather than trying to construct a logically consistent object from the file.

From my perspective, the namelist is malformed, and the behavior is undefined. But if you can construct a Fortran program which successfully parses &key on some compiler, then let's see what can be done.

If not, then I think we should probably regard this as a bug, and should try to signal to the user that this namelist cannot be parsed correctly.

[kripnerl]

I am not aware of the compiler. However, this was used as a configuration file which was part of OMFIT suite (https://gafusion.github.io/OMFIT-source/). I suppose that the configuration file may later be served to the compiled program. But it can be as well only an internal representation of the OMFIT. I am not sure though. Nevertheless, I wanted to point it out as it may interest you.

Thank you for the provided info.

[kripnerl]

edit: I've edited the expected output of my example. Since it was wrong.

[kripnerl]

I will let you know. If I later find the usage of this namelist in the actual file.

[marshallward]

Thanks for pointing it out to me. There was another project (#42, #43) which had also repurposed the namelist syntax. We tried to include support, but there was ultimately too much conflict between the tokens and we abandoned it.

This looks like something similar, since the project seems to be used to manage several model simulations. However, I cannot even find the source on github (This link appears to be broken.) so it's hard to confirm the details.

Even though this would not be as difficult, I think I learned my lesson from SDSS which is that it's better to do one format well, rather than try to incorporate any format which might resemble a namelist.

It is interesting to know about this usage of namelist syntax though, thanks again for linking it here. It may be worthwhile to reach out to the OMFIT maintainers to see if they're interested in finding a way to support their syntax.

[kripnerl]

You can reach some code here (https://gafusion.github.io/OMFIT-source/code.html#module-classes.omfit_namelist). GA Github is probably closed for people outside the community.

I will notify the relevant people if they are willing to resolve this discussion.

[smithsp]

@marshallward I certainly agree with your statement "it's better to do one format well,". While the OMFIT namelist class can handle a nested structure, it does not handle 2d arrays well. If you would like access to the OMFIT source code, just let me know, and I will add you to the private repo (without permission you get a 404 error). Maybe someday we will get OMFIT to be open source.
Mention @orso82 @zhucaoxiang and https://github.com/gafusion/OMFIT-source/issues/3031

[orso82]

The OMFIT namelist parser is designed to be more robust than what a strict fortran namelist parser would do, and handle the typical errors that users would do. Fortran parsers themselves can handle some of these issues, albeit different compilers may handle some things differently. The OMFIT namelist class can collate array data in a single python object, dump the array data in different formats, access the data using fortran conventions, and handle sparse arrays. Lastly, this class can also do some things that are not standard for a fortran namelist, such as nested namelists.

@kripnerl FYI, if it helps you can use OMFIT classes outside of the whole framework as described here: https://omfit.io/run.html#using-omfit-as-a-python-library

[marshallward]

Thanks very much for replying to this, @smithsp and @orso82 and sorry for the slow response.

Although the namelist syntax is non-compliant, it may be possible to silently add support for this. Currently f90nml doesn't do enough to catch incorrect syntax, and often it just creates unpredictable results, as seen above. It's not crazy to just call this "undefined behavior" which produces results favorable to OMFIT.

I am not sure if this is what we want. Ideally I would prefer a stronger parser which rejects invalid namelists. Nor am I sure it is wise to explicitly add support for non-compliant namelists. But I am open to the idea.

Ideally I'd prefer to redesign f90nml so that someone could import it and alter its componets to conform with a new format. But that may be an even bigger task.

I will sit on this for now, since it's not really something I can afford to work on, but will update here if I manage to get anywhere.

[marshallward]

Unfortunately I think that current limitations on my time prevent me from seriously considering support for namelist-adjacent formats, so I will have to close this one.

Perhaps someday the parser could be written as a grammer with permitted tweaks by the user, but that is so far off that I think that in the meantime this goal must be abandoned.