Add ability to read old-style metadata to v1.5

[achilleas-k]

Old style metadata (Value object) should be readable in the new library. The new file format should be upgraded to 1.2.0. Library-file compatibility would then be as follows:

• NIX 1.4.x will be unable to read 1.2.0 files
• NIX 1.5.x will be able to read and write 1.2.0 files
• NIX 1.5.x will be be able to read (not write) 1.1.1 files

Multivalue properties in 1.1.1 (old) files should be returned as a vector of raw values. The uncertainty property should be accessible through a separate method. In case the Value objects in the file have different values for uncertainty, a warning should be raised.

[matham]

I'm wondering what this change means for how I use nixpy. I have been using 1.4.x and all my files are in it. Most of the people in my lab are also on 1.4.x.

I wanted to try 1.5.x so I installed the most recent 1.5.x beta. However, it immediately choked trying to load the files created on 1.4.x because of this change. Specifically, in 1.4.x I read a property with prop.values[0].value, but with 1.5.x I need to do prop.values[0].

My question is,

• if I create a file in 1.5.x, would I be able to read it in 1.4.x? I'm guessing not.
• if I create a file in 1.4.x, presumably once this issue is fixed I'll be able to read it in 1.5.x?
• For me it seems like creating a special function which reads the property depending on the nix version is really the only option to be able to support both 1.5.x and 1.4.x!?
• Is there a changelog somewhere that I could look for other breaking changes between versions? I tried looking for one, but couldn't find it.

[jgrewe]

Hi @matham, you are right, that the 1.4.x libraries will not be able to read files created with 1.5.x libraries. The 1.5.x libraries will be able to read (but not write) files written with 1.4.x libs. We do not maintain a dedicated changelog (but probably should do so). Changes are always indicated in the release notes, though. We are not done with the 1.5 version yet (still beta). We will make sure the release notes contain the needed information (in particular, the breaking ones). There will be a converter script.

• With 1.5 we introduce DataFrames that allow storing spreadsheet like data.
• The c++ bindings were dropped. nixpy is then a pure python package.
• We further simplified the metadata part of the nix data model to unify it with the odml data model. This also simplifies handling the values, they are lists then.

The change you indicated should be mostly all that needs to be done when reading 1.4 files. Would you be fine with this? Do you have special demands? Please let us know and we will be happy to make the change as comfortable as possible.

[matham]

Thanks for clarifying these points. The new DataFrame and being a pure python package sounds amazing.

For the reading part, I made this functions:

def read_nix_prop(prop):
    try:
        return prop.values[0].value
    except AttributeError:
        return prop.values[0][0]

which seems to work well. Maybe it'd be wroth having such a function in the library in a compat module or I guess the docs? Anyway, this part is fine now for me.

The other issue I just ran into is that I have this (and similar) lines f.sections['app_logs']['notes'] += notes, where notes is a string. I had made the file in 1.4.x and was trying to update it in 1.5.x. This obviously failed as you mentioned.

My question is now, is there a way to be able to upgrade a file into 1.5.x? Because my strategy would then be as follows when I need to update a file on a different system than the one it was written in:

• If it was written in 1.5.x and is now updated on 1.4.x, raise an exception to indicate that the user should upgrade to 1.5.x.
• If it was written in 1.4.x and is now updated on 1.5.x, upgrade the file (hopefully in place if possible, or through a copy) to 1.5.x and then update the data.

The simplest way on my end would be to walk all the properties, delete them, and set it again. Would this work? Could this be added as a function to 1.5.x?

[jgrewe]

You could also check the file format version instead of catching the exception.

f = nix.File.open(filename, nix.FileMode.ReadOnly)
print("file: %s \n\tformat: %s \n\tformat version: %s \n\tlibrary version %s"
          % (filename, f.format, f.version, nix.__version__))
f.close()

tbh, I am not fully up to date regarding the converter (maybe @achilleas-k can be drawn into the discussion :)). I agree with your first point, raising an exception and pointing to the new library version seems plausible. The second one is trickier, from my point of view. As far as I remember we considered creating a copy and not to do an in-place conversion, but it could be worth to at least offer this option.

In our lab, we basically face the same problem, so far everything is written with the 1.4 version, but I will switch, once the 1.5 c++ lib is done (mainly because of the DataFrames, which will simplify the file structures considerably). Our way of working with the files, however, is mainly a ReadOnly access so I think I will not convert them on the fly, but maybe in a dedicated effort at some point.

Thank you for your input and thoughts! Please feel free to contact us or create issues if you're missing certain functionality.