Conflict between extension neurodata type property and core neurodata type property

[ehennestad]

The linked extension schema has a couple of neurodata types that extends Device, with the definition of a model property:

https://github.com/histedlab/ndx-photostim/blob/main/spec/ndx-photostim.extensions.yaml

With the upcoming 2.9.0 release, and the introduction of the Device.model being a DeviceModel, there might be some conflicts that could arise. For example, in MatNWB the export logic will cause an error when exporting a device from this extension with NWB v2.9.0.

Are there any guidelines for what do in a situation like this?

[oruebel]

I'm wondering whether the extension could specify compliance restrictions, e.g., requires NWB <2.9 to avoid creating new files that are not compatible or update the extension for NWB 2.9

@rly thoughts?

[rly]

@oruebel @stephprince and I discussed this issue earlier this week. Thanks for bringing up this use case. It is rare but should be addressed.

The short answer is that the extension is not compatible with the newer core schema and the extension developers will need to update their extension to be compatible (or if we support it, then they could more simply mark their extension as compatible with core version restrictions). Until then, users who want to write data using the extension will have to use an older version of the schema.

For write:

For MatNWB, since the API is generated, this is doable, but it is not clear which versions are compatible (unless we support specifying that in the schema and the developers add the restriction). An error should be raised when loading an extension that is incompatible with the core. I suppose MatNWB can try to generate APIs for schemas moving backwards in time until one is compatible.

For PyNWB, the latest version of PyNWB only writes with the cached version of the core schema. This means that writing an older version of the core schema requires downgrading the version of PyNWB. An error should be raised when loading an extension that is incompatible with the core (this is not yet implemented).

For read:

For MatNWB, I believe the API is generated from the cached schema. Is that right? Then there should not be an issue.

For PyNWB, the API will expect that Device.model is a link to a DeviceModel. Because the data says that DeviceExt.model is a string, this will raise an error. We could improve this behavior by mangling the name of the DeviceExt.model attribute to be DeviceExt.ndx_photostim__model so that the data is still accessible. However, this will still confuse any downstream tools and scripts that expect a string value in DeviceExt.model. Those tools and scripts will have to update to not use the latest PyNWB, or change their behavior to account for the change in PyNWB.

These changes will take some time to implement in a general way, but I don't think it would be too bad. I'll sketch out the algorithm for evaluating compatibility of schema. What do you all think?