iris
iris copied to clipboard
SciTools
CF global attribute "Conventions": expose to users, and include CF standard name table version
✨ Feature Request
Currently, the global attribute Conventions is difficult to change. The default is hardcoded deep in the code:
https://github.com/SciTools/iris/blob/021d6ff7f719c600a189cfa5671bf32d20f66853/lib/iris/fileformats/netcdf/saver.py#L104 and changing the default is not as easy as one (at least I) would have hoped for, because it is "hidden" as iris.config.netcdf.conventions_override = True, which is difficult to find if I search for "global attribute Conventions" (or similar) in the documentation.
For our own use cases we would like to have better control of which CF version is actually written into the files, and moreover we would like to include additional conventions, like ACDD, Clix-meta version, as well as the CF Standard name Table version. Hence, this feature request has three main components:
- In
iris.saveallow users to pass the kwargConventions='....'. - Change the
tools/generate_std_names.pyso that thelib/std_names.pyincludes the version number (and possibly related information such as creation date). - New functionality (in
iris.util?) to get the default Conventions string, and to get the Standard name table version number fromlib/st_names.py.
Motivation
A more general motivation for giving the user (easier) access to setting the CF version is given by CF itself in Section 2.6.1:
When CF is listed with other conventions, this asserts the same full compliance with CF requirements and interpretations as if CF was the sole convention. It is the responsibility of the data-writer to ensure that all common metadata is used with consistent meaning between conventions.
Not to speak against the proposal, but I just thought it would be worth explaining here why we control this at all, and what it really means -- which I don't think is written down anywhere in user docs ...
In my view, at least, the only meaning that you can attach to the CF version which Iris writes in output files is "whatever we put in this file should be CF-compliant up to this CF version" :
Effectively, that means "the file does not include any CF encoding from CF versions later than this" -- and that's really all.
You can probably infer that an application interpreting all aspects of that CF version can understand everything that we have tried to state about the data. But it doesn't (even) mean that the output is definitely valid under later CF versions -- though to date I think CF have totally avoided any backwards-invalidating changes. Only that it may contain information which is not understood by earlier versions.
Yes, I understand this and do agree that this is a very reasonable Iris developer perspective. But from our side, as users of iris and building applications using the Iris api the perspective is a slightly different one. The files we produce never make use of all CF/Iris mechanisms, and most of them are fairly standard in most respects, which means that they are equally valid in later CF versions. But occasionally there are improvements and clarifications, and even corrections to the CF Conventions text that either is taken on board by Iris, or handled by us, which means that it is meaningful to indicate this in the files. Or that we have projects or users that want the data to follow some specific CF version.
The main motivation behind the request is however that we want to append additional conventions to the Conventions string, and then we have no easy/nice/clean way to get the Iris default string to start with. Add to this the other part of the request, that we would like to have access to the Standard name table version.
as users of iris and building applications using the Iris api the perspective is a slightly different one.
Thanks, understood !
I have been working in this area lately, in the context of https://github.com/SciTools/iris/issues/3325 and the project for that . ( which hopefully will make it into Iris 3.6 ) The way Iris handles Conventions attributes is certainly a bit odd + we probably want to fix some things. I will take care how this might interact with that work.
#5423 records the table version in the header of iris.std_names, and a public global variable iris.std_names.CF_STANDARD_NAME_TABLE_VERSION.
@SciTools/peloton We need to discuss this further amongst ourselves, but in the meantime, have you thought about ncdata for post-save modification?
Good news: we discovered a secret feature :godmode: !
This gets used very rarely, and isn't even documented, so it has faded from developers' memory, but it is there. Please let us know if it works for you 😊
iris.site_configuration is designed to modify certain aspects of Iris based on a config file. This allows for a more global modification versus an argument in a function:
- A permanent setting for a piece of software downstream of Iris
- Can share a single config file between a team
- Even if just writing scripts: don't need identical lines in multiple calls
Modifying Conventions was one of the imagined use cases, since it would often be something like an institution-wide phrase that needs to be added.
Permanent setting - create a file: iris/site_config.py
def cf_patch_conventions(conventions: str) -> str:
return f"{conventions} extra" if conventions else conventions
def update(config: dict) -> None:
config["cf_profile"] = lambda cube: None
config["cf_patch_conventions"] = cf_patch_conventions
Expand for how this works
https://github.com/SciTools/iris/blob/46d2cf6f684d06fae2933a4110f888c12afa2e68/lib/iris/init.py#L249-L258
Temporary setting - modify iris.site_configuration
import iris
def cf_patch_conventions(conventions: str) -> str:
return f"{conventions} extra" if conventions else conventions
iris.site_configuration = {
"cf_profile": lambda cube: None,
"cf_patch_conventions": cf_patch_conventions,
}
iris.save(something, "something.nc")
You could even put this into a context manager
from contextlib import contextmanager
import iris
@contextmanager
def convention_addition(addition: str) -> None:
def cf_patch_conventions(conventions: str) -> str:
return f"{conventions} {addition}" if conventions else conventions
original_config = iris.site_configuration.copy()
iris.site_configuration = {
"cf_profile": lambda cube: None,
"cf_patch_conventions": cf_patch_conventions,
}
yield
iris.site_configuration = original_config
with convention.addition("extra):
iris.save(something, "something.nc")
More info
All modifications are currently just for Iris saving. This feature is currently undocumented so if you'd like to know the full extent of what you can do please browse the code at the below points, and perhaps search the Iris tests:
https://github.com/SciTools/iris/blob/46d2cf6f684d06fae2933a4110f888c12afa2e68/lib/iris/fileformats/netcdf/saver.py#L563
https://github.com/SciTools/iris/blob/46d2cf6f684d06fae2933a4110f888c12afa2e68/lib/iris/fileformats/netcdf/saver.py#L649-L652
https://github.com/SciTools/iris/blob/46d2cf6f684d06fae2933a4110f888c12afa2e68/lib/iris/fileformats/netcdf/saver.py#L2780-L2782
