1.00.27 branch

[aradhakrishnanGFDL]

It appears that the ESGF publisher expects a branch/tag corresponding to the data request version as found in the global attribute of the netCDF files. Since, we at GFDL are attempting to publishing a dataset that adheres to 1.00.27 data request, it looks like the publisher is failing to check out the corresponding version of the cmor tables since there is no matching branch or tag for 1.00.27. Please let us know if we are missing something.

Note: We use this tag for testing https://github.com/PCMDI/cmip6-cmor-tables/releases/tag/6.2.11.2 outside of the ESGF workflow. Should this also be in a branch named after the data request version to satisfy the publisher needs?

Please let us know how to handle this. Thank you.

[taylor13]

I'm afraid I don't understand the publisher well enough to know how it is supposed to work. As you probably know, a single released version of the CMOR tables is based on a single version of the CV's hosted by PCMDI and a single version of the data request.

The most recent CMOR table release (6.3.27-fixed, August 1, 2018) is based on version 6.2.11.2 of the CV's and a slightly corrected ("fixed") version of data request 1.00.27. The correction made to release 1.00.27 can be seen at https://github.com/PCMDI/cmip6-cmor-tables/commit/56018b5fd49937424ccaacfa76799f32d26aa1a6#diff-651ae5789ab776f2fcaf80b5e62801dc .

There have been updates to the CV's hosted at PCMDI since the 6.2.11.2 release, but I think the only one that affects the publisher is discussed in https://github.com/WCRP-CMIP/CMIP6_CVs/issues/592 which lists the experiments affected. The updated release is more lenient in which model components are "allowed" for a few experiments.

I don't think we can expect the publisher to be able to automatically handle all combinations of data request releases and CV versions. I hope it is possible for that publication manager to specify a directory where the appropriate CMIP6_CV.json file and the appropriate and the appropriate variable tables can be found and relied on by PrePARE to check the submitted datasets.

I'll work with the ESGF folks and get them to advise you on this issue.

[taylor13]

I was planning to speak with Sasha today about this, but he is on travel until next week. Assuming the default tables accessed by the publisher (based on "data_specs_version") can be overridden by the person publishing the data, you should obtain the CMOR tables in release 3.6.27 (or 3.6.27-fixed) and you should replace the CMIP6_CVs.json file with the latest one (I would wait until a new "nightly" version is available ... maybe tomorrow). That will have any changes made since the CV version used in 3.6.27.

[sashakames]

It does appear that a 01.00.27 tag is missing. That will probably solve GFDLs problem. I misunderstood the issue, sorry.

[aradhakrishnanGFDL]

Hi @sashakames, Thank you. That should resolve one of the issues. We hope to be able to give publishing another try in the coming week after the creation of this tag in the CMOR repo.

The second issue that was alluded to in this GitHub issue is: how we can point to the most recent version of CMIP6_CV (to make sure the correct source and activity participations are accounted for in case there are changes made to the CV later on. We presently have this situation :( ) and continue to use this specific 01.00.27 tag for data request/cmip6 tables. We like Karl's idea of..

"I hope it is possible for that publication manager to specify a directory where the appropriate CMIP6_CV.json file and the appropriate and the appropriate variable tables can be found and relied on by PrePARE to check the submitted datasets."

Thoughts?

[doutriaux1]

@aradhakrishnanGFDL we should have a cronjob back up and running so changes to CV will go almsot instantaneously back into master from now on.

[taylor13]

to be sure, multiple CMOR/PrePARE CMIP6_CV.json tables are associated with a single data request release (such as 01.00.27), so tagging a CMIP6_CV.json with 01.00.27 may not get you to the correct CMIP6_CV.json. (The difference between the several CMIP6_CV.json files associated with a single data request is that "institution_id"s and "source_id"s are continually updated (even after a a given data request release, and the newer institution_id's and source_id's may be needed by PrePARE). Only the data data request release number is recorded in CMIP6 netCDF files, not the version number for the CMIP6_CV.json (because only those using CMOR need access to CMIP6_CV.json when writing their files; of course the publisher does need to access CMIP6_CV.json, but that comes later). So, the publisher can't really know in all cases what version of CMIP6_CV.json it should use, given just the information in the netCDF file; the person publishing may need to tell it where to find the CMIP6_CV.json that is appropriate for his/her data.

Hope this is somewhat clear.

[sashakames]

The publisher will automatically update the local copy of CMIP6_CV.json with the latest verson from "master" regardless of which data_specs were used. This, essentially creates a "detached" hybrid branch with the tables from the specified data_specs_version and the .json file. This was the feature needed by IPSL and implemented by @glevava

[taylor13]

oh, that's pretty cool. So the publisher will generate a new CMIP6_CV.json, drawing on the one tagged, e.g., 01.00.27, but modifying the part of that .json that is taken from the CMIP6_source_id.json and CMIP6_institution_id.json? Or does it just look in master and find the latest CMIP6_CV.json file?

[glevava]

The CMIP6_CV.json file of CMOR repo is an aggregation of all CMIP6_*.json from WCRP repo. PrePARE expect to check global attributes from netCDF files against the "latest" version of the CMIP6_CV.json (i.e., the last JSON from WCRP). At the beginning of Data Challenges we had errors regarding the further-info-url syntax for instance, the syntaxe of the DR 01.00.21 (i.e., with http instead of https) was not accepted by PrePARE even using the appropriate table tag. Some tests into CMOR have be hard coded and require the latest version CMIP6_CV.json. Now, what the publisher does is to fetch first the JSON tables from the appropriate branch of the CMOR Github. In a second step, it fetch the CMIP6_CV.json (only) from the master overwriting the CMIP6_CV.json initially fetched from the tag/branch driving by the data_specs_version.

I hope I was clear... : /

FYI, at IPSL we didn't use this "git mechanism". We fetch the tables using "esgfetchtables" in a "subdirectories" way. This avoid some strange errors GFDL and I encountered during DC.

[taylor13]

Got it (I think). you end up with the .json files that contain data request information from the CMOR table release tagged with the number retrieved from "data_specs_version". But you use the WCRP CV information from the master (which is updated nightly??).

Sorry to have been slow. @doutriaux1: Has a "01.00.27" tag been assigned yet in the CMOR repo? I think we need one on 6.3.27 in https://github.com/PCMDI/cmip6-cmor-tables/releases , but maybe someone (@aradhakrishnanGFDL ?) following this thread should confirm first.

[aradhakrishnanGFDL]

Hi @doutriaux1 and @taylor13, Thank you. We use 6.3.27-fixed for 01.00.27 currently. https://github.com/PCMDI/cmip6-cmor-tables/tree/b427077cbca2c28b2948752c63e4d3f68449fc1f https://github.com/PCMDI/cmip6-cmor-tables/releases/tag/6.2.11.2

[durack1]

@aradhakrishnanGFDL it appears this is an old stale issue that was solve years ago, closing - please reopen if it's something that continues to cause headaches