Wishlist/Recommendation: "Downstream testing" against NWB-Extensions

[yarikoptic]

NWB extensions have their lives, and NWB-schema/HDMF/PyNWB and those extensions as well, can progress forward while possibly missing the point whenever a change in any of those caused some files containing the extensions not even loadable (#1337, #1335).

To help (if not to eliminate) to identify and fix similar issues early in the process, and to make development of pynwb aligned with the ecosystem of extensions it would be great if per each extension there were

• sample nwb files with a given extension, possibly multiple files to correspond to various versions of the extension/base pynwb which produced them

and have a setup where

• all extensions are tested against current released as well as against "in development" hdmf/PyNWB
• rudimentary smoke (or + some basic unittests for expected data structures etc) test of PyNWB to load sample files mentioned above

PS feel free to refile somewhere against extensions, but as pynwb in my view is the entry point for users, it should also be pynwb's "duty" to CI against extensions on regular basis if not in each PR. Eg. in datalad we do test against only a limited subset of extensions within PRs (https://github.com/datalad/datalad/actions?query=workflow%3AExtensions) and then across all of the extensions in a dedicated repo/dashboard: https://github.com/datalad/datalad-extensions/#datalad-extensions-status . May be some of those ideas/setups come handy (datalad-extensions setup should be quite generalizable - workflows are templated from a single spec)

[oruebel]

I agree that having testing infrastructure for extensions will be useful. I'm not sure, whether this will need to be part of the extension catalog, individual extensions, or PyNWB (probably some combination of things). I thin this would be a good topic for the upcoming hackathon to do some specification on.